flexbackup FAQ

# flexbackup frequently-asked questions

Suggestions/additions welcome

FAQ Revised: Friday 10 October 2003 11:26:37

## Table of Contents

1. Configuration
  - 1.1. How do I upgrade from 1.0.x to 1.2.x?
  - 1.2. Why won't my exclude or prune expressions work?
  - 1.3. How can I avoid running backups or remote shells as root?
  - 1.4. On my system, "tar" is not GNU tar but is the vendor's "tar". GNU tar is installed as "gtar". On my system, "dump" is "ufsdump".
  - 1.5. How can I back up Windows systems?
  - 1.6. Does the to-disk backup mode use single archive files or directory tree copies of files?
  - 1.7. How do I deal with spaces in set or prune directory specifications?
  - 1.8. Does flexbackup work with autoloaders / mtx?
2. Backup
  - 2.1. Can you explain numeric dump levels?
  - 2.2. Do you support multi-tape archives?
  - 2.3. I'm backing up to files on disk -- How can I get filenames to not have timestamps in them?
  - 2.4. What is the maxiumum size of the archives flexbackup can produce?
3. Recovery / Extraction
  - 3.1. How do I extract only certain files in an archive rather than the whole thing?
  - 3.2. How do I find out which archive(s) contain a certain file?
  - 3.3. The disk totally died. How do I get flexbackup's archives off the tapes on a machine without flexbackup or even perl installed?
  - 3.4. I don't have the index files! (disk died, different machine, etc.)
4. Common problems
  - 4.1. I get I/O errors or other errors from my tape drive.
  - 4.2. Backups seem to work ok, but I can't get extracting/listing to work.
  - 4.3. How do I get remote backups over ssh to work without prompting for a password?
  - 4.4. I keep getting only one backup per tape when I do "-set all"!
  - 4.5. 'buffer' complains that it can't run successfully or that it doesn't have enough shared memory.
  - 4.6. When using -newtape or -erase, the tape drive sits there forever making noise and nothing happens on the screen
  - 4.7. My "mt" or tape drive doesn't support operation xxxx.
  - 4.8. Flexbackup caused my system to hang!
  - 4.9. The backup looks like it works ok, but flexbackup complains of an error at the end of the archive, then exits.
  - 4.10. I get an error from the backup due to files moving/changing/being deleted during the backup
5. Misc
  - 5.1. How does the "packaging system delta" (or "-pkgdelta") feature work?
  - 5.2. What exactly is the "-wday" switch for?
  - 5.3. How can I tell flexbackup to read from stdin or backup to stdout?
  - 5.4. Why keep the table of contents db on disk rather than on the tape?
  - 5.5. This index stuff is a useless mess/I can't get it to work!

## 1. Configuration

1.1. How do I upgrade from 1.0.x to 1.2.x?
  Use the new default config file, then tweak as needed. Specific things that changed:
  1. Filesystems: the filesystem spec now allows an arbitrary "backup set" concept. In the config file, you can do things like:

               $set{'tag1'} = "/dir1 /dir2 ...";
               $set{'tag2'} = "/dir3 host:/dir4 ...";

  2. The commandline -fs switch became:
     - "-dir <dir>" backs up one directory tree (old "-fs <dir>")
     - "-set <tag>" backs up a named set. Useful new feature.
     - "-set all" acts like old "-fs all". If using tapes & level 0, each set is a different tape (acts like the old numeric index to @filesystems)
  3. mt blocksize: old configuration boolean $mt_var_blksize is now an integer, $mt_blocksize (in bytes).
     - Set to 0 for variable blocksize - old "true" value
     - Comment out or set to "$blksize * 1024" for old "false" value.
     - Or use some other value, like "512". This wasn't possible before.
  4. "-extract -files" turned into "-extract -onefile" or "-extract -flist" (functionality should be more clear now).
1.2. Why won't my exclude or prune expressions work?
  Use regular expressions, not wild cards (aka globs). See a perl, sed, or grep reference, or "man 7 regex". Simplistically, you probably mean to say ".*" instead of "*", and ".?" instead of "?". Also, leave OFF the first part of a path that is the filesystem you are archiving; the exclude expressions only match sub-paths of the backup directory.

  You can test exclusion, levels, pruning, and package deltas by using "-type filelist". This writes a text file listing all files that would have been backed up into your current working directory.

  You can also test to see if your expression matches what you want to exclude by:
  `cd <topdir>; find . -regex "<expression>"`

1.3. How can I avoid running backups or remote shells as root?
  You can do this with 'sudo'.
  1. Set up a passphraseless ssh key for a user to run the backups.
  2. Set `$path{command} = 'sudo command';` in the config file. Do this for all commands that will need privilege (find, afio/tar/etc)
  3. Set up an /etc/sudoers file with something like this:
     `Cmnd_Alias BACKUP = /usr/bin/afio, /usr/bin/find
     backupuser ALL = NOPASSWD: BACKUP`
1.4. On my system, "tar" is not GNU tar but is the vendor's "tar". GNU tar is installed as "gtar". On my system, "dump" is "ufsdump".
  Use the path overrides at the end of the config file. A couple examples are given there, such as: `$path{'dump'} = 'ufsdump';`. You can use this mechanism to override paths or just change command names.
1.5. How can I back up Windows systems?
  There are currently two ways:
  - Use "smbmount" to mount the windows share on a directory and back that up just like any other directory
  - flexbackup runs natively under cygwin

  In the future we might add a way to set up "smbtar" or something similar...

1.6. Does the to-disk backup mode use single archive files or directory tree copies of files?
  You can do either. The "copy" or "rsync" types will do a tree mirror of the selected files, other types make single-file archives in the `$device` dir.

  Note that archive-to-disk with remote directories (`$device = host:/path/to/dir`) is not currently possible. You need to either run the backup from the machine with the disk, or use NFS.

1.7. How do I deal with spaces in set or prune directory specifications?
  Good question, since these variables are set up as space-separated lists. Just enclose each item in single or double quotes:
  `$set{'backup'} = "'/path/file system 1' '/path/file system 2'";`
1.8. Does flexbackup work with autoloaders / mtx?
  Not currently. If you want help add it, let us know. See also 2.2 below.

## 2. Backup

2.1. Can you explain numeric dump levels?
  Level 0 gets everything. For 1 or higher, files that have been changed since the last backup at a lower level are archived. For instance, if a level 2 backup was done on Monday, followed by a level 4 backup on Tuesday, a subsequent level 3 backup on Wednesday would contain all files modified or added since the level 2 (Monday) backup.

  See also these sun docs for an example with some pictures.

  Traditional Unix 'dump' behavior only allows levels 0-9; flexbackup removes this limit if you are using any of the non-dump archive formats.

  For those people coming from other platforms that are not used to dump-style numeric levels, the latest versions of flexbackup have added `-full`, `-differential`, `-incremental` that get translated into appropriate dump-style level numbers. Definitions:

  - full: everything (duh!)
  - differential: only files that have been changed or added since the last full backup
  - incremental: only the files that have changed or been added to a system since doing the last backup (of any type)

  These are simply aliased to numeric levels: full = 0, differential = 1, and incremental = (latest backup + 1)

2.2. Do you support multi-tape archives?
  If you split filesystems/directories to be in different "sets", each set will go on a different tape for full backups (you'll be prompted).

  As of now, having one directory archive span multiple tapes is not supported. True, some of the archiver programs can deal with end-of media if and only if they are writing directly to the device, but it can't be done in a shell pipeline, or in any kind of generic fashion. For now, split the directory up into workable portions, use larger media, or backup to disk...

  We need a program like "buffer", but that also deals with the volume end detection & media change prompting, has configuraable "new tape" scripts, etc.

  Version 1.2.1 has basic support for using mbuffer with multivolumes. Please try it out and let me know.

2.3. I'm backing up to files on disk -- How can I get filenames to not have timestamps in them?
  Set `$staticfiles` to true in the config file. Archives will then have names such as "etc.3.dump.gz" rather than something like "home.0.200301212305.afio-bz2". There is a similar parameter for the log files, by the way.
2.4. What is the maxiumum size of the archives flexbackup can produce?
  This is a limitation of the archive program, tape driver, underlying filesystem, OS, or specific compilation of perl - nothing in flexbackup controls that.

  Some older programs/systems barf at 2GB files -- anything recent should deal with large files no problem, using standard APIs.

  If you are using "buffer" be aware that you need to compile with extra flags, or patch the 1.19 version to handle >2GB files. (The RPMS linked to from this site have been)

  See also Large File Support in Linux or Large file size changes to the single Unix spec for more info and hints on how to fix things if you need to recompile something.

## 3. Recovery / Extraction

3.1. How do I extract only certain files in an archive rather than the whole thing?
  Use the -flist or -onefile switches.

  To quickly extract just a single file, use "-extract -onefile path/to/my/file", giving the path from the archive.

  To extract a list of multiple files, put them into a text file, for instance "restorelist", then use "-extract -flist restorelist". The format is one line per pathname, using the path of the file in the archive. Note if you are using afio with compression you need to append ".z" to filenames for any compressed files (depends on threshold and exclusion patterns).

3.2. How do I find out which archive(s) contain a certain file?
  If you don't know in which archive to find a certain file, look at the log files. As long as you have verbose turned on (default), you can just 'zgrep filename /var/log/flexbackup/*.gz', and that works pretty well.
3.3. The disk totally died. How do I get flexbackup's archives off the tapes on a machine without flexbackup or even perl installed?
  Something like this, if using compression with tar/cpio/dump:
  `dd if=/dev/tapedevice bs=10k | gzip -dc | tar xvf -`
  replace "tar xvf -" with "restore -i -f -" or whatever. Skip the gzip pipe if you aren't using compression. afio is a bit more complex, you might need something like:
  `afio -i -k -x -P bzip2 -Q -d -Z -v`
  Check the manpages for the archive utility you used...
3.4. I don't have the index files! (disk died, different machine, etc.)
  This is OK, you can still extract/restore/etc. The index files are for human convenience only. Just skip the first file on the tape, it's a simple "tag" text file. The rest of the tape is your normal archives.

## 4. Common problems

4.1. I get I/O errors or other errors from my tape drive.
  Run this: `flexbackup -test-tape-drive`. It writes a couple small files to the tape, then reads & diffs them. This will flush out any problems with parameters like blocking, filemarks, and padding; or issues with the tape drive or driver itself.

  If it fails, try these one at a time:

  1. If you are using buffering, turn it off (`$buffer='false'`) until you can get things working without it.
  2. Change the block size parameter (`$blksize`) in the configuration file. The default is 10k, but some tape drives like 32k or 64k much better.
  3. Try setting `$pad_blocks` to false
  4. Switch the setting of `$mt_blocksize`.
  5. Try with with `$indexes` set to false

  If you are using an IDE tape drive under Linux, and are still having trouble, you might also try the 'ide-scsi' layer with the 'st' module, and treating the thing like a SCSI tape - I've had more success that way.

  You can also just try writing/reading stuff straight from the drive with an archiver and dd (to factor flexbackup out of the loop), although that's what the `-test-tape-drive` switch tries to do. If you can get that working, but flexbackup still seems to malfunction, let us know. Try things like: (The example below uses afio to test, tar/cpio/others will be different)

        # Backup two dirs
        mt -f /dev/tape setblk 32768
        mt -f /dev/tape rewind
        mt -f /dev/tape erase
        find /dir1 -print | afio -o -z -v -b 32k - | dd ibs=32k obs=32k of=/dev/tape
        find /dir2 -print | afio -o -z -v -b 32k - | dd ibs=32k obs=32k of=/dev/tape

        # List
        mt -f /dev/tape rewind
        dd ibs=32k obs=32k if=/dev/tape | afio -t -z -v -b 32k -
        dd ibs=32k obs=32k if=/dev/tape | afio -t -z -v -b 32k -

4.2. Backups seem to work ok, but I can't get extracting/listing to work.
  Usually a matter of tape positioning, or else is a tape drive parameter problem as talked about above. For positioning, use 'mt rewind; mt fsf <n>' to get to the right file number on the tape, or use the "-num" switch with "-list|extract|compare|restore" to have flexbackup do it for you. If you are using flexbackup's indexes, remember that file number 0 is a header used for the index itself.
4.3. How do I get remote backups over ssh to work without prompting for a password?
  For rsh, use .rhosts or hosts.equiv. But you really should NOT be using rsh.

  If you are using ssh and are having a problem with it asking for a password, then you don't have a passphrase-less authorized key set up, your RSAAuthentication parameters are incorrect, or you have some other public/private key problem. Section 7.2.2 of the ssh FAQ might help:
  http://www.onsight.com/faq/ssh/ssh-faq.html
  In a nutshell the value of your `~/.ssh/identity.pub` needs to be present in your `~/.ssh/authorized_keys` file for RSA authentication to work. For root accounts you might also need to put
  `PermitRootLogin yes`
  in your sshd_config file, as root is treated with more care than normal user accounts. Note: you might choose without-password or forced-commands-only instead of yes for more security. See your ssh(1) and sshd(1) man pages.

  If you do not want to use passphrase-less ssh keys with root logins, See the section on use with "sudo". (Or else sit at the terminal and type passwords during the backups if you *really* want to...)

4.4. I keep getting only one backup per tape when I do "-set all"!
  Use the non-rewinding device.
4.5. 'buffer' complains that it can't run successfully or that it doesn't have enough shared memory.
  Set `$buffer_megs` to a lower value and try again. I usually run with it set somewhere between 5-20MB.

  On FreeBSD, SysV shared memory can run low using the GENERIC kernel at its default value (especially if you are running something like GNOME). You can modify your kernel file and change the line
  `options SHMMAXPGS=2048`

  To something really big. You can also change this dynamically at startup by putting
  `kern.ipc.shmmax={large integer}`

  in /etc/sysctl.conf. Other BSD's will have very similar mechanisms.

4.6. When using -newtape or -erase, the tape drive sits there forever making noise and nothing happens on the screen
  Some types of tape drives/drivers take an extremely long time to do an "mt erase" operation, or its unnecessary. If you suspect this is the case, check to see if "mt erase" process is running. In these cases, you can set the config variable `$erase_rewind_only` to true - then an "erase" operation becomes just a rewind. If an archive is then written, filemarks and tape end-of-data should work fine.

  Also, this hang can be caused by incorrect tape drive parameters on some systems. Try "flexbackup -test-tape-drive". If it hangs, and there is a dd or buffer process continuously running, kill it and adjust blocksizes and/or padding.

4.7. My "mt" or tape drive doesn't support operation xxxx.
  mt commands definitely vary greatly from platform to platform. We try to pick the best default, but it won't be right for everyone. To override mt operation 'xxxx', just put $mt{'xxxx'} = 'yyyy' in the config (see the comments near the end). For instance, to make flexbackup use "mt rdspos" instead of "mt rdhpos, use:
  `$mt{'rdhpos'} = 'rdspos';`
4.8. Flexbackup caused my system to hang!
  No it didn't. Give me a break, it's a script. If you are asking this question most likely your tape driver lost its mind and hung the system somehow.
4.9. The backup looks like it works ok, but flexbackup complains of an error at the end of the archive, then exits.
  If flexbackup detects a non-zero exit status for any command in the pipeline, it will stop. Something probably happened further up in the log that caused the archive or buffering program to exit with an error status. Look over the log again, or run with "-d verbose=false" to turn off the printing of filenames to make it more obvious. See also the next question.
4.10. I get an error from the backup due to files moving/changing/being deleted during the backup
  You really shouldn't be doing that!

  If flexbackup detects a non-zero exit status for any command in the pipeline, it will stop. You can override this with the "--ignore-errors" switch, but then it will just blindly continue on such cases. You have been warned.

  Some archivers are more picky than others. You can try switching the archive type.

  If it is something like a SQL database data file causing the problem, you should dump the database to a secondary location and back THAT data up. (See the mysqldump command, for instance).

## 5. Misc

5.1. How does the "packaging system delta" (or "-pkgdelta") feature work?
  When you have OS CD's sitting around, and/or you know darn well that 90% of the non-user, non-data files on your system you can get off the 'net easily, there is no need to back up "stock" files.

  Flexbackup can use a "package delta" mode that cuts down the number of files to be archived by basically doing the following:

  1. List the files in the directory/filesystem to be backed up, using the normal level & timstamp mechanism
  2. Subtract any files that are "owned" by packages.
  3. Add back any package files modified from the distributed version (optional)
  4. Backup the files in the list.

  It was orginally intended for use with RPM-based systems ("-pkgdelta rpm"). Initial support for FreeBSD packages is included ("-pkgdelta freebsd"), although there the "base" files aren't part of package so they can't be excluded.

  If someone knows how to do the equivalent things for .deb packages or others, let me know and we can add it.

5.2. What exactly is the "-wday" switch for?
  To help cron do things like once-a-month, certain day of the week jobs. With crontab fields, you can easily set up something to run on certain day(s) of the week, or on certain day(s) of the month.

  Now, tell me you how to set up a cron job to run once a month on the first Sunday only.... you can't do it.

  The "-wday" flag tells flexbackup to just exit if the current day of the week does not match the flag. Now, to fulfill the above requirement, you add a cron entry like:

         0 3 1-7 * * flexbackup -wday 7 -set all ....

  Which will run at 3:00AM on the first Sunday of every month.
5.3. How can I tell flexbackup to read from stdin or backup to stdout?
  See the "-pipe" switch. Only lets you do one directory at a time, obviously. You can use this to interact with other programs in a pipeline.
5.4. Why keep the table of contents db on disk rather than on the tape?
  - We don't know the table size ahead of time unless we use a hard-limited reserved size.
  - Updating the db at the beginning of the tape will add tape-travel overhead for every backup.
  - a backup->rewind->write start of tape sequence will lose the rest of the contents past the first file unless we start to play filemark tricks.
5.5. This index stuff is a useless mess/I can't get it to work!
  It isn't necessary if you don't like it. Set `$indexes = 'false';` in the config file, and just use written labels on the tapes...

FAQ compilation thanks to makefaq