External Filters, Programs and Commands

Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev |  | Next

# Chapter 16. External Filters, Programs and Commands

Standard UNIX commands make shell scripts more versatile. The power of scripts comes from coupling system commands and shell directives with simple programming constructs.

# 16.1. Basic Commands

The first commands a novice learns

ls

  The basic file "list" command. It is all too easy to underestimate the power of this humble command. For example, using the `-R`, recursive option, ls provides a tree-like listing of a directory structure. Other useful options are `-S`, sort listing by file size, `-t`, sort by file modification time, `-v`, sort by (numerical) version numbers embedded in the filenames, [1] `-b`, show escape characters, and `-i`, show file inodes (see Example 16-4).

       bash$ ls -l
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter10.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter11.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter12.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter1.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter2.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter3.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:49 Chapter_headings.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:49 Preface.txt

       bash$ ls -lv
        total 0
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:49 Chapter_headings.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:49 Preface.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter1.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter2.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter3.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter10.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter11.txt
       -rw-rw-r-- 1 bozo bozo 0 Sep 14 18:44 chapter12.txt

  | The ls command returns a non-zero exit status when attempting to list a non-existent file.
       bash$ ls abc
       ls: abc: No such file or directory

       bash$ echo $?
       2

  Example 16-1. Using ls to create a table of contents for burning a CDR disk

         1 #!/bin/bash
         2 # ex40.sh (burn-cd.sh)
         3 # Script to automate burning a CDR.
         4
         5
         6 SPEED=10         # May use higher speed if your hardware supports it.
         7 IMAGEFILE=cdimage.iso
         8 CONTENTSFILE=contents
         9 # DEVICE=/dev/cdrom     For older versions of cdrecord
        10 DEVICE="1,0,0"
        11 DEFAULTDIR=/opt  # This is the directory containing the data to be burned.
        12                  # Make sure it exists.
        13                  # Exercise: Add a test for this.
        14
        15 # Uses Joerg Schilling's "cdrecord" package:
        16 # http://www.fokus.fhg.de/usr/schilling/cdrecord.html
        17
        18 #  If this script invoked as an ordinary user, may need to suid cdrecord
        19 #+ chmod u+s /usr/bin/cdrecord, as root.
        20 #  Of course, this creates a security hole, though a relatively minor one.
        21
        22 if [ -z "$1" ]
        23 then
        24   IMAGE_DIRECTORY=$DEFAULTDIR
        25   # Default directory, if not specified on command-line.
        26 else
        27     IMAGE_DIRECTORY=$1
        28 fi
        29
        30 # Create a "table of contents" file.
        31 ls -lRF $IMAGE_DIRECTORY > $IMAGE_DIRECTORY/$CONTENTSFILE
        32 # The "l" option gives a "long" file listing.
        33 # The "R" option makes the listing recursive.
        34 # The "F" option marks the file types (directories get a trailing /).
        35 echo "Creating table of contents."
        36
        37 # Create an image file preparatory to burning it onto the CDR.
        38 mkisofs -r -o $IMAGEFILE $IMAGE_DIRECTORY
        39 echo "Creating ISO9660 file system image ($IMAGEFILE)."
        40
        41 # Burn the CDR.
        42 echo "Burning the disk."
        43 echo "Please be patient, this will take a while."
        44 wodim -v -isosize dev=$DEVICE $IMAGEFILE
        45 #  In newer Linux distros, the "wodim" utility assumes the
        46 #+ functionality of "cdrecord."
        47 exitcode=$?
        48 echo "Exit code = $exitcode"
        49
        50 exit $exitcode

cat, tac

  cat, an acronym for concatenate, lists a file to `stdout`. When combined with redirection (> or >>), it is commonly used to concatenate files.

         1 # Uses of 'cat'
         2 cat filename                          # Lists the file.
         3
         4 cat file.1 file.2 file.3 > file.123   # Combines three files into one.

  The `-n` option to cat inserts consecutive numbers before all lines of the target file(s). The `-b` option numbers only the non-blank lines. The `-v` option echoes nonprintable characters, using ^ notation. The `-s` option squeezes multiple consecutive blank lines into a single blank line.

  See also Example 16-28 and Example 16-24.

  | In a pipe, it may be more efficient to redirect the `stdin` to a file, rather than to cat the file.
         1 cat filename | tr a-z A-Z
         2
         3 tr a-z A-Z < filename   #  Same effect, but starts one less process,
         4                         #+ and also dispenses with the pipe.

  tac, is the inverse of cat, listing a file backwards from its end.

rev

  reverses each line of a file, and outputs to `stdout`. This does not have the same effect as tac, as it preserves the order of the lines, but flips each one around (mirror image).

       bash$ cat file1.txt
       This is line 1.
       This is line 2.

       bash$ tac file1.txt
       This is line 2.
       This is line 1.

       bash$ rev file1.txt
       .1 enil si sihT
       .2 enil si sihT

cp

  This is the file copy command. `cp file1 file2` copies `file1` to `file2`, overwriting `file2` if it already exists (see Example 16-6).

  | Particularly useful are the `-a` archive flag (for copying an entire directory tree), the `-u` update flag (which prevents overwriting identically-named newer files), and the `-r` and `-R` recursive flags.
         1 cp -u source_dir/* dest_dir
         2 #  "Synchronize" dest_dir to source_dir
         3 #+  by copying over all newer and not previously existing files.

mv

  This is the file move command. It is equivalent to a combination of cp and rm. It may be used to move multiple files to a directory, or even to rename a directory. For some examples of using mv in a script, see Example 10-11 and Example A-2.

  | When used in a non-interactive script, mv takes the `-f` (force) option to bypass user input. When a directory is moved to a preexisting directory, it becomes a subdirectory of the destination directory.
       bash$ mv source_directory target_directory

       bash$ ls -lF target_directory
       total 1
       drwxrwxr-x    2 bozo  bozo      1024 May 28 19:20 source_directory/

rm

  Delete (remove) a file or files. The `-f` option forces removal of even readonly files, and is useful for bypassing user input in a script.

  | The rm command will, by itself, fail to remove filenames beginning with a dash. Why? Because rm sees a dash-prefixed filename as an option.
       bash$ rm -badname
       rm: invalid option -- b
       Try `rm --help' for more information.
  One clever workaround is to precede the filename with a " -- " (the end-of-options flag).
       bash$ rm -- -badname
  Another method to is to preface the filename to be removed with a `dot-slash` .
       bash$ rm ./-badname

  | When used with the recursive flag `-r`, this command removes files all the way down the directory tree from the current directory. A careless rm -rf * can wipe out a big chunk of a directory structure.

rmdir

  Remove directory. The directory must be empty of all files -- including "invisible" dotfiles [2] -- for this command to succeed.

mkdir

  Make directory, creates a new directory. For example, `mkdir -p project/programs/December` creates the named directory. The `-p` option automatically creates any necessary parent directories.

chmod

  Changes the attributes of an existing file or directory (see Example 15-14).

         1 chmod +x filename
         2 # Makes "filename" executable for all users.
         3
         4 chmod u+s filename
         5 # Sets "suid" bit on "filename" permissions.
         6 # An ordinary user may execute "filename" with same privileges as the file's owner.
         7 # (This does not apply to shell scripts.)

         1 chmod 644 filename
         2 #  Makes "filename" readable/writable to owner, readable to others
         3 #+ (octal mode).
         4
         5 chmod 444 filename
         6 #  Makes "filename" read-only for all.
         7 #  Modifying the file (for example, with a text editor)
         8 #+ not allowed for a user who does not own the file (except for root),
         9 #+ and even the file owner must force a file-save
        10 #+ if she modifies the file.
        11 #  Same restrictions apply for deleting the file.

         1 chmod 1777 directory-name
         2 #  Gives everyone read, write, and execute permission in directory,
         3 #+ however also sets the "sticky bit".
         4 #  This means that only the owner of the directory,
         5 #+ owner of the file, and, of course, root
         6 #+ can delete any particular file in that directory.
         7
         8 chmod 111 directory-name
         9 #  Gives everyone execute-only permission in a directory.
        10 #  This means that you can execute and READ the files in that directory
        11 #+ (execute permission necessarily includes read permission
        12 #+ because you can't execute a file without being able to read it).
        13 #  But you can't list the files or search for them with the "find" command.
        14 #  These restrictions do not apply to root.
        15
        16 chmod 000 directory-name
        17 #  No permissions at all for that directory.
        18 #  Can't read, write, or execute files in it.
        19 #  Can't even list files in it or "cd" to it.
        20 #  But, you can rename (mv) the directory
        21 #+ or delete it (rmdir) if it is empty.
        22 #  You can even symlink to files in the directory,
        23 #+ but you can't read, write, or execute the symlinks.
        24 #  These restrictions do not apply to root.

chattr

  Change file attributes. This is analogous to chmod above, but with different options and a different invocation syntax, and it works only on ext2/ext3 filesystems.

  One particularly interesting chattr option is `i`. A chattr +i `filename` marks the file as immutable. The file cannot be modified, linked to, or deleted, not even by root. This file attribute can be set or removed only by root. In a similar fashion, the `a` option marks the file as append only.

       root# chattr +i file1.txt

       root# rm file1.txt

       rm: remove write-protected regular file `file1.txt'? y
       rm: cannot remove `file1.txt': Operation not permitted

  If a file has the `s` (secure) attribute set, then when it is deleted its block is overwritten with binary zeroes. [3]

  If a file has the `u` (undelete) attribute set, then when it is deleted, its contents can still be retrieved (undeleted).

  If a file has the `c` (compress) attribute set, then it will automatically be compressed on writes to disk, and uncompressed on reads.

  | The file attributes set with chattr do not show in a file listing (ls -l).

ln

  Creates links to pre-existings files. A "link" is a reference to a file, an alternate name for it. The ln command permits referencing the linked file by more than one name and is a superior alternative to aliasing (see Example 4-6).

  The ln creates only a reference, a pointer to the file only a few bytes in size.

  The ln command is most often used with the `-s`, symbolic or "soft" link flag. Advantages of using the `-s` flag are that it permits linking across file systems or to directories.

  The syntax of the command is a bit tricky. For example: `ln -s oldfile newfile` links the previously existing `oldfile` to the newly created link, `newfile`.

  | If a file named `newfile` has previously existed, an error message will result.

  Links give the ability to invoke a script (or any other type of executable) with multiple names, and having that script behave according to how it was invoked.

  Example 16-2. Hello or Good-bye

         1 #!/bin/bash
         2 # hello.sh: Saying "hello" or "goodbye"
         3 #+          depending on how script is invoked.
         4
         5 # Make a link in current working directory ($PWD) to this script:
         6 #    ln -s hello.sh goodbye
         7 # Now, try invoking this script both ways:
         8 # ./hello.sh
         9 # ./goodbye
        10
        11
        12 HELLO_CALL=65
        13 GOODBYE_CALL=66
        14
        15 if [ $0 = "./goodbye" ]
        16 then
        17   echo "Good-bye!"
        18   # Some other goodbye-type commands, as appropriate.
        19   exit $GOODBYE_CALL
        20 fi
        21
        22 echo "Hello!"
        23 # Some other hello-type commands, as appropriate.
        24 exit $HELLO_CALL

man, info

  These commands access the manual and information pages on system commands and installed utilities. When available, the info pages usually contain more detailed descriptions than do the man pages.

  There have been various attempts at "automating" the writing of man pages. For a script that makes a tentative first step in that direction, see Example A-39.

### Notes

[1] | The `-v` option also orders the sort by upper- and lowercase prefixed filenames.
[2] | Dotfiles are files whose names begin with a dot, such as `~/.Xdefaults`. Such filenames do not appear in a normal ls listing (although an ls -a will show them), and they cannot be deleted by an accidental rm -rf *. Dotfiles are generally used as setup and configuration files in a user's home directory.
[3] | This particular feature may not yet be implemented in the version of the ext2/ext3 filesystem installed on your system. Check the documentation for your Linux distro.

Prev | Home | Next
Internal Commands and Builtins | Up | Complex Commands