Here Documents

Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev |  | Next

# Chapter 19. Here Documents

| Here and now, boys. --Aldous Huxley, Island

A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command, such as ftp, cat, or the ex text editor.

       1 COMMAND <<InputComesFromHERE
       2 ...
       3 ...
       4 ...
       5 InputComesFromHERE

A limit string delineates (frames) the command list. The special symbol << precedes the limit string. This has the effect of redirecting the output of a command block into the `stdin` of the program or command. It is similar to `interactive-program < command-file`, where `command-file` contains

       1 command #1
       2 command #2
       3 ...

The here document equivalent looks like this:

       1 interactive-program <<LimitString
       2 command #1
       3 command #2
       4 ...
       5 LimitString

Choose a limit string sufficiently unusual that it will not occur anywhere in the command list and confuse matters.

Note that here documents may sometimes be used to good effect with non-interactive utilities and commands, such as, for example, wall.

Example 19-1. broadcast: Sends message to everyone logged in

       1 #!/bin/bash
       2
       3 wall <<zzz23EndOfMessagezzz23
       4 E-mail your noontime orders for pizza to the system administrator.
       5     (Add an extra dollar for anchovy or mushroom topping.)
       6 # Additional message text goes here.
       7 # Note: 'wall' prints comment lines.
       8 zzz23EndOfMessagezzz23
       9
      10 # Could have been done more efficiently by
      11 #         wall <message-file
      12 #  However, embedding the message template in a script
      13 #+ is a quick-and-dirty one-off solution.
      14
      15 exit

Even such unlikely candidates as the vi text editor lend themselves to here documents.

Example 19-2. dummyfile: Creates a 2-line dummy file

       1 #!/bin/bash
       2
       3 # Noninteractive use of 'vi' to edit a file.
       4 # Emulates 'sed'.
       5
       6 E_BADARGS=85
       7
       8 if [ -z "$1" ]
       9 then
      10   echo "Usage: `basename $0` filename"
      11   exit $E_BADARGS
      12 fi
      13
      14 TARGETFILE=$1
      15
      16 # Insert 2 lines in file, then save.
      17 #--------Begin here document-----------#
      18 vi $TARGETFILE <<x23LimitStringx23
      19 i
      20 This is line 1 of the example file.
      21 This is line 2 of the example file.
      22 ^[
      23 ZZ
      24 x23LimitStringx23
      25 #----------End here document-----------#
      26
      27 #  Note that ^[ above is a literal escape
      28 #+ typed by Control-V <Esc>.
      29
      30 #  Bram Moolenaar points out that this may not work with 'vim'
      31 #+ because of possible problems with terminal interaction.
      32
      33 exit

The above script could just as effectively have been implemented with ex, rather than vi. Here documents containing a list of ex commands are common enough to form their own category, known as ex scripts.

       1 #!/bin/bash
       2 #  Replace all instances of "Smith" with "Jones"
       3 #+ in files with a ".txt" filename suffix.
       4
       5 ORIGINAL=Smith
       6 REPLACEMENT=Jones
       7
       8 for word in $(fgrep -l $ORIGINAL *.txt)
       9 do
      10   # -------------------------------------
      11   ex $word <<EOF
      12   :%s/$ORIGINAL/$REPLACEMENT/g
      13   :wq
      14 EOF
      15   # :%s is the "ex" substitution command.
      16   # :wq is write-and-quit.
      17   # -------------------------------------
      18 done

Analogous to "ex scripts" are cat scripts.

Example 19-3. Multi-line message using cat

       1 #!/bin/bash
       2
       3 #  'echo' is fine for printing single line messages,
       4 #+  but somewhat problematic for for message blocks.
       5 #   A 'cat' here document overcomes this limitation.
       6
       7 cat <<End-of-message
       8 -------------------------------------
       9 This is line 1 of the message.
      10 This is line 2 of the message.
      11 This is line 3 of the message.
      12 This is line 4 of the message.
      13 This is the last line of the message.
      14 -------------------------------------
      15 End-of-message
      16
      17 #  Replacing line 7, above, with
      18 #+   cat > $Newfile <<End-of-message
      19 #+       ^^^^^^^^^^
      20 #+ writes the output to the file $Newfile, rather than to stdout.
      21
      22 exit 0
      23
      24
      25 #--------------------------------------------
      26 # Code below disabled, due to "exit 0" above.
      27
      28 # S.C. points out that the following also works.
      29 echo "-------------------------------------
      30 This is line 1 of the message.
      31 This is line 2 of the message.
      32 This is line 3 of the message.
      33 This is line 4 of the message.
      34 This is the last line of the message.
      35 -------------------------------------"
      36 # However, text may not include double quotes unless they are escaped.

The `-` option to mark a here document limit string (`<<-LimitString`) suppresses leading tabs (but not spaces) in the output. This may be useful in making a script more readable.

Example 19-4. Multi-line message, with tabs suppressed

       1 #!/bin/bash
       2 # Same as previous example, but...
       3
       4 #  The - option to a here document <<-
       5 #+ suppresses leading tabs in the body of the document,
       6 #+ but *not* spaces.
       7
       8 cat <<-ENDOFMESSAGE
       9 	This is line 1 of the message.
      10 	This is line 2 of the message.
      11 	This is line 3 of the message.
      12 	This is line 4 of the message.
      13 	This is the last line of the message.
      14 ENDOFMESSAGE
      15 # The output of the script will be flush left.
      16 # Leading tab in each line will not show.
      17
      18 # Above 5 lines of "message" prefaced by a tab, not spaces.
      19 # Spaces not affected by   <<-  .
      20
      21 # Note that this option has no effect on *embedded* tabs.
      22
      23 exit 0

A here document supports parameter and command substitution. It is therefore possible to pass different parameters to the body of the here document, changing its output accordingly.

Example 19-5. Here document with replaceable parameters

       1 #!/bin/bash
       2 # Another 'cat' here document, using parameter substitution.
       3
       4 # Try it with no command-line parameters,   ./scriptname
       5 # Try it with one command-line parameter,   ./scriptname Mortimer
       6 # Try it with one two-word quoted command-line parameter,
       7 #                           ./scriptname "Mortimer Jones"
       8
       9 CMDLINEPARAM=1     #  Expect at least command-line parameter.
      10
      11 if [ $# -ge $CMDLINEPARAM ]
      12 then
      13   NAME=$1          #  If more than one command-line param,
      14                    #+ then just take the first.
      15 else
      16   NAME="John Doe"  #  Default, if no command-line parameter.
      17 fi
      18
      19 RESPONDENT="the author of this fine script"
      20
      21
      22 cat <<Endofmessage
      23
      24 Hello, there, $NAME.
      25 Greetings to you, $NAME, from $RESPONDENT.
      26
      27 # This comment shows up in the output (why?).
      28
      29 Endofmessage
      30
      31 # Note that the blank lines show up in the output.
      32 # So does the comment.
      33
      34 exit

This is a useful script containing a here document with parameter substitution.

Example 19-6. Upload a file pair to Sunsite incoming directory

       1 #!/bin/bash
       2 # upload.sh
       3
       4 #  Upload file pair (Filename.lsm, Filename.tar.gz)
       5 #+ to incoming directory at Sunsite/UNC (ibiblio.org).
       6 #  Filename.tar.gz is the tarball itself.
       7 #  Filename.lsm is the descriptor file.
       8 #  Sunsite requires "lsm" file, otherwise will bounce contributions.
       9
      10
      11 E_ARGERROR=85
      12
      13 if [ -z "$1" ]
      14 then
      15   echo "Usage: `basename $0` Filename-to-upload"
      16   exit $E_ARGERROR
      17 fi
      18
      19
      20 Filename=`basename $1`           # Strips pathname out of file name.
      21
      22 Server="ibiblio.org"
      23 Directory="/incoming/Linux"
      24 #  These need not be hard-coded into script,
      25 #+ but may instead be changed to command-line argument.
      26
      27 Password="your.e-mail.address"   # Change above to suit.
      28
      29 ftp -n $Server <<End-Of-Session
      30 # -n option disables auto-logon
      31
      32 user anonymous "$Password"       #  If this doesn't work, then try:
      33                                  #  quote user anonymous "$Password"
      34 binary
      35 bell                             # Ring 'bell' after each file transfer.
      36 cd $Directory
      37 put "$Filename.lsm"
      38 put "$Filename.tar.gz"
      39 bye
      40 End-Of-Session
      41
      42 exit 0

Quoting or escaping the "limit string" at the head of a here document disables parameter substitution within its body. The reason for this is that quoting/escaping the limit string effectively escapes the $, `, and \ special characters, and causes them to be interpreted literally. (Thank you, Allen Halsey, for pointing this out.)

Example 19-7. Parameter substitution turned off

       1 #!/bin/bash
       2 #  A 'cat' here-document, but with parameter substitution disabled.
       3
       4 NAME="John Doe"
       5 RESPONDENT="the author of this fine script"
       6
       7 cat <<'Endofmessage'
       8
       9 Hello, there, $NAME.
      10 Greetings to you, $NAME, from $RESPONDENT.
      11
      12 Endofmessage
      13
      14 #   No parameter substitution when the "limit string" is quoted or escaped.
      15 #   Either of the following at the head of the here document would have
      16 #+  the same effect.
      17 #   cat <<"Endofmessage"
      18 #   cat <<\Endofmessage
      19
      20
      21
      22 #   And, likewise:
      23
      24 cat <<"SpecialCharTest"
      25
      26 Directory listing would follow
      27 if limit string were not quoted.
      28 `ls -l`
      29
      30 Arithmetic expansion would take place
      31 if limit string were not quoted.
      32 $((5 + 3))
      33
      34 A a single backslash would echo
      35 if limit string were not quoted.
      36 \\
      37
      38 SpecialCharTest
      39
      40
      41 exit

Disabling parameter substitution permits outputting literal text. Generating scripts or even program code is one use for this.

Example 19-8. A script that generates another script

       1 #!/bin/bash
       2 # generate-script.sh
       3 # Based on an idea by Albert Reiner.
       4
       5 OUTFILE=generated.sh         # Name of the file to generate.
       6
       7
       8 # -----------------------------------------------------------
       9 # 'Here document containing the body of the generated script.
      10 (
      11 cat <<'EOF'
      12 #!/bin/bash
      13
      14 echo "This is a generated shell script."
      15 #  Note that since we are inside a subshell,
      16 #+ we can't access variables in the "outside" script.
      17
      18 echo "Generated file will be named: $OUTFILE"
      19 #  Above line will not work as normally expected
      20 #+ because parameter expansion has been disabled.
      21 #  Instead, the result is literal output.
      22
      23 a=7
      24 b=3
      25
      26 let "c = $a * $b"
      27 echo "c = $c"
      28
      29 exit 0
      30 EOF
      31 ) > $OUTFILE
      32 # -----------------------------------------------------------
      33
      34 #  Quoting the 'limit string' prevents variable expansion
      35 #+ within the body of the above 'here document.'
      36 #  This permits outputting literal strings in the output file.
      37
      38 if [ -f "$OUTFILE" ]
      39 then
      40   chmod 755 $OUTFILE
      41   # Make the generated file executable.
      42 else
      43   echo "Problem in creating file: \"$OUTFILE\""
      44 fi
      45
      46 #  This method also works for generating
      47 #+ C programs, Perl programs, Python programs, Makefiles,
      48 #+ and the like.
      49
      50 exit 0

It is possible to set a variable from the output of a here document. This is actually a devious form of command substitution.

       1 variable=$(cat <<SETVAR
       2 This variable
       3 runs over multiple lines.
       4 SETVAR
       5 )
       6
       7 echo "$variable"

A here document can supply input to a function in the same script.

Example 19-9. Here documents and functions

       1 #!/bin/bash
       2 # here-function.sh
       3
       4 GetPersonalData ()
       5 {
       6   read firstname
       7   read lastname
       8   read address
       9   read city
      10   read state
      11   read zipcode
      12 } # This certainly appears to be an interactive function, but . . .
      13
      14
      15 # Supply input to the above function.
      16 GetPersonalData <<RECORD001
      17 Bozo
      18 Bozeman
      19 2726 Nondescript Dr.
      20 Bozeman
      21 MT
      22 21226
      23 RECORD001
      24
      25
      26 echo
      27 echo "$firstname $lastname"
      28 echo "$address"
      29 echo "$city, $state $zipcode"
      30 echo
      31
      32 exit 0

It is possible to use : as a dummy command accepting output from a here document. This, in effect, creates an "anonymous" here document.

Example 19-10. "Anonymous" Here Document

       1 #!/bin/bash
       2
       3 : <<TESTVARIABLES
       4 ${HOSTNAME?}${USER?}${MAIL?}  # Print error message if one of the variables not set.
       5 TESTVARIABLES
       6
       7 exit $?

| A variation of the above technique permits "commenting out" blocks of code.

Example 19-11. Commenting out a block of code

       1 #!/bin/bash
       2 # commentblock.sh
       3
       4 : <<COMMENTBLOCK
       5 echo "This line will not echo."
       6 This is a comment line missing the "#" prefix.
       7 This is another comment line missing the "#" prefix.
       8
       9 &*@!!++=
      10 The above line will cause no error message,
      11 because the Bash interpreter will ignore it.
      12 COMMENTBLOCK
      13
      14 echo "Exit value of above \"COMMENTBLOCK\" is $?."   # 0
      15 # No error shown.
      16 echo
      17
      18
      19 #  The above technique also comes in useful for commenting out
      20 #+ a block of working code for debugging purposes.
      21 #  This saves having to put a "#" at the beginning of each line,
      22 #+ then having to go back and delete each "#" later.
      23 #  Note that the use of of colon, above, is optional.
      24
      25 echo "Just before commented-out code block."
      26 #  The lines of code between the double-dashed lines will not execute.
      27 #  ===================================================================
      28 : <<DEBUGXXX
      29 for file in *
      30 do
      31  cat "$file"
      32 done
      33 DEBUGXXX
      34 #  ===================================================================
      35 echo "Just after commented-out code block."
      36
      37 exit 0
      38
      39
      40
      41 ######################################################################
      42 #  Note, however, that if a bracketed variable is contained within
      43 #+ the commented-out code block,
      44 #+ then this could cause problems.
      45 #  for example:
      46
      47
      48 #/!/bin/bash
      49
      50   : <<COMMENTBLOCK
      51   echo "This line will not echo."
      52   &*@!!++=
      53   ${foo_bar_bazz?}
      54   $(rm -rf /tmp/foobar/)
      55   $(touch my_build_directory/cups/Makefile)
      56 COMMENTBLOCK
      57
      58
      59 $ sh commented-bad.sh
      60 commented-bad.sh: line 3: foo_bar_bazz: parameter null or not set
      61
      62 # The remedy for this is to strong-quote the 'COMMENTBLOCK' in line 49, above.
      63
      64   : <<'COMMENTBLOCK'
      65
      66 # Thank you, Kurt Pfeifle, for pointing this out.

| Yet another twist of this nifty trick makes "self-documenting" scripts possible.

Example 19-12. A self-documenting script

       1 #!/bin/bash
       2 # self-document.sh: self-documenting script
       3 # Modification of "colm.sh".
       4
       5 DOC_REQUEST=70
       6
       7 if [ "$1" = "-h"  -o "$1" = "--help" ]     # Request help.
       8 then
       9   echo; echo "Usage: $0 [directory-name]"; echo
      10   sed --silent -e '/DOCUMENTATIONXX$/,/^DOCUMENTATIONXX$/p' "$0" |
      11   sed -e '/DOCUMENTATIONXX$/d'; exit $DOC_REQUEST; fi
      12
      13
      14 : <<DOCUMENTATIONXX
      15 List the statistics of a specified directory in tabular format.
      16 ---------------------------------------------------------------
      17 The command-line parameter gives the directory to be listed.
      18 If no directory specified or directory specified cannot be read,
      19 then list the current working directory.
      20
      21 DOCUMENTATIONXX
      22
      23 if [ -z "$1" -o ! -r "$1" ]
      24 then
      25   directory=.
      26 else
      27   directory="$1"
      28 fi
      29
      30 echo "Listing of "$directory":"; echo
      31 (printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \
      32 ; ls -l "$directory" | sed 1d) | column -t
      33
      34 exit 0

Using a cat script is an alternate way of accomplishing this.

       1 DOC_REQUEST=70
       2
       3 if [ "$1" = "-h"  -o "$1" = "--help" ]     # Request help.
       4 then                                       # Use a "cat script" . . .
       5   cat <<DOCUMENTATIONXX
       6 List the statistics of a specified directory in tabular format.
       7 ---------------------------------------------------------------
       8 The command-line parameter gives the directory to be listed.
       9 If no directory specified or directory specified cannot be read,
      10 then list the current working directory.
      11
      12 DOCUMENTATIONXX
      13 exit $DOC_REQUEST
      14 fi

See also Example A-28, Example A-40, Example A-41, and Example A-42 for more examples of self-documenting scripts.

| Here documents create temporary files, but these files are deleted after opening and are not accessible to any other process.
     bash$ bash -c 'lsof -a -p $$ -d0' << EOF
     > EOF
     lsof    1213 bozo    0r   REG    3,5    0 30386 /tmp/t1213-0-sh (deleted)

| Some utilities will not work inside a here document.

| The closing limit string, on the final line of a here document, must start in the first character position. There can be no leading whitespace. Trailing whitespace after the limit string likewise causes unexpected behavior. The whitespace prevents the limit string from being recognized. [1]
       1 #!/bin/bash
       2
       3 echo "----------------------------------------------------------------------"
       4
       5 cat <<LimitString
       6 echo "This is line 1 of the message inside the here document."
       7 echo "This is line 2 of the message inside the here document."
       8 echo "This is the final line of the message inside the here document."
       9      LimitString
      10 #^^^^Indented limit string. Error! This script will not behave as expected.
      11
      12 echo "----------------------------------------------------------------------"
      13
      14 #  These comments are outside the 'here document',
      15 #+ and should not echo.
      16
      17 echo "Outside the here document."
      18
      19 exit 0
      20
      21 echo "This line had better not echo."  # Follows an 'exit' command.

| Some people very cleverly use a single ! as a limit string. But, that's not necessarily a good idea.
       1 # This works.
       2 cat <<!
       3 Hello!
       4 ! Three more exclamations !!!
       5 !
       6
       7
       8 # But . . .
       9 cat <<!
      10 Hello!
      11 Single exclamation point follows!
      12 !
      13 !
      14 # Crashes with an error message.
      15
      16
      17 # However, the following will work.
      18 cat <<EOF
      19 Hello!
      20 Single exclamation point follows!
      21 !
      22 EOF
      23 # It's safer to use a multi-character limit string.

For those tasks too complex for a here document, consider using the `expect` scripting language, which was specifically designed for feeding input into interactive programs.

# 19.1. Here Strings

> A here string can be considered as a stripped-down form of a here document.
> It consists of nothing more than COMMAND <<< $WORD,
> where `$WORD` is expanded and fed to the `stdin` of COMMAND.

As a simple example, consider this alternative to the echo-grep construction.

       1 # Instead of:
       2 if echo "$VAR" | grep -q txt   # if [[ $VAR = *txt* ]]
       3 # etc.
       4
       5 # Try:
       6 if grep -q "txt" <<< "$VAR"
       7 then   #         ^^^
       8    echo "$VAR contains the substring sequence \"txt\""
       9 fi
      10 # Thank you, Sebastian Kaminski, for the suggestion.

Or, in combination with read:

       1 String="This is a string of words."
       2
       3 read -r -a Words <<< "$String"
       4 #  The -a option to "read"
       5 #+ assigns the resulting values to successive members of an array.
       6
       7 echo "First word in String is:    ${Words[0]}"   # This
       8 echo "Second word in String is:   ${Words[1]}"   # is
       9 echo "Third word in String is:    ${Words[2]}"   # a
      10 echo "Fourth word in String is:   ${Words[3]}"   # string
      11 echo "Fifth word in String is:    ${Words[4]}"   # of
      12 echo "Sixth word in String is:    ${Words[5]}"   # words.
      13 echo "Seventh word in String is:  ${Words[6]}"   # (null)
      14                                                  # Past end of $String.
      15
      16 # Thank you, Francisco Lobo, for the suggestion.

It is, of course, possible to feed the output of a here string into the `stdin` of a loop.

       1 # As Seamus points out . . .
       2
       3 ArrayVar=( element0 element1 element2 {A..D} )
       4
       5 while read element ; do
       6   echo "$element" 1>&2
       7 done <<< $(echo ${ArrayVar[*]})
       8
       9 # element0 element1 element2 A B C D

Example 19-13. Prepending a line to a file

       1 #!/bin/bash
       2 # prepend.sh: Add text at beginning of file.
       3 #
       4 #  Example contributed by Kenny Stauffer,
       5 #+ and slightly modified by document author.
       6
       7
       8 E_NOSUCHFILE=85
       9
      10 read -p "File: " file   # -p arg to 'read' displays prompt.
      11 if [ ! -e "$file" ]
      12 then   # Bail out if no such file.
      13   echo "File $file not found."
      14   exit $E_NOSUCHFILE
      15 fi
      16
      17 read -p "Title: " title
      18 cat - $file <<<$title > $file.new
      19
      20 echo "Modified file is $file.new"
      21
      22 exit  # Ends script execution.
      23
      24   from 'man bash':
      25   Here Strings
      26   	A variant of here documents, the format is:
      27
      28   		<<<word
      29
      30   	The word is expanded and supplied to the command on its standard input.
      31
      32
      33   Of course, the following also works:
      34    sed -e '1i\
      35    Title: ' $file

Example 19-14. Parsing a mailbox

       1 #!/bin/bash
       2 #  Script by Francisco Lobo,
       3 #+ and slightly modified and commented by ABS Guide author.
       4 #  Used in ABS Guide with permission. (Thank you!)
       5
       6 # This script will not run under Bash versions -lt 3.0.
       7
       8
       9 E_MISSING_ARG=87
      10 if [ -z "$1" ]
      11 then
      12   echo "Usage: $0 mailbox-file"
      13   exit $E_MISSING_ARG
      14 fi
      15
      16 mbox_grep()  # Parse mailbox file.
      17 {
      18     declare -i body=0 match=0
      19     declare -a date sender
      20     declare mail header value
      21
      22
      23     while IFS= read -r mail
      24 #         ^^^^                 Reset $IFS.
      25 #  Otherwise "read" will strip leading & trailing space from its input.
      26
      27    do
      28        if [[ $mail =~ ^From  ]]   # Match "From" field in message.
      29        then
      30           (( body  = 0 ))           # "Zero out" variables.
      31           (( match = 0 ))
      32           unset date
      33
      34        elif (( body ))
      35        then
      36             (( match ))
      37             #  echo "$mail"
      38             #  Uncomment above line if you want entire body
      39             #+ of message to display.
      40
      41    elif [[ $mail ]]; then
      42       IFS=: read -r header value <<< "$mail"
      43       #                          ^^^  "here string"
      44
      45       case "$header" in
      46       [Ff][Rr][Oo][Mm] ) [[ $value =~ "$2" ]] && (( match++ )) ;;
      47       # Match "From" line.
      48       [Dd][Aa][Tt][Ee] ) read -r -a date <<< "$value" ;;
      49       #                                  ^^^
      50       # Match "Date" line.
      51       [Rr][Ee][Cc][Ee][Ii][Vv][Ee][Dd] ) read -r -a sender <<< "$value" ;;
      52       #                                                    ^^^
      53       # Match IP Address (may be spoofed).
      54       esac
      55
      56        else
      57           (( body++ ))
      58           (( match  )) &&
      59           echo "MESSAGE ${date:+of: ${date[*]} }"
      60        #    Entire $date array             ^
      61           echo "IP address of sender: ${sender[1]}"
      62        #    Second field of "Received" line    ^
      63
      64        fi
      65
      66
      67     done < "$1" # Redirect stdout of file into loop.
      68 }
      69
      70
      71 mbox_grep "$1"  # Send mailbox file to function.
      72
      73 exit $?
      74
      75 # Exercises:
      76 # ---------
      77 # 1) Break the single function, above, into multiple functions,
      78 #+   for the sake of readability.
      79 # 2) Add additional parsing to the script, checking for various keywords.
      80
      81
      82
      83 $ mailbox_grep.sh scam_mail
      84   MESSAGE of Thu, 5 Jan 2006 08:00:56 -0500 (EST)
      85   IP address of sender: 196.3.62.4

Exercise: Find other uses for here strings, such as, for example, feeding input to dc.

### Notes

[1] | Except, as Dennis Benzinger points out, if using <<- to suppress tabs.

Prev | Home | Next
Globbing | Up | I/O Redirection