Analog and web server logfile rollovers

# HOW-TO
Rotate Analog and web server logfiles

This HOW-TO written by Brian Clifton with thanks to Karel Kerezman
Originally written 2000-06-29. Last update 2002-01-04.

## Purpose

Analog is claimed as the most popular web logfile analyser in the world. (Details). Whether running multiple virtual hosts or a single root web server, a useful feature is to run Analog and then roll over both the logfile just analysed and the Analog report file. At the same time, the logfile can be compressed and the Analog report e-mailed to yourself or the virtual host client. This can be achieved quite simply using crontab and logrotate.

## System

This example was developed and tested using a default install of RedHat 6.1 using Apache v1.3.9-8 and Analog v4.11. Also running on RedHat 7.0 using Apache 1.3.14. Please note, this is just one example and is not the only method of achieving the same goal!

## Schematic example

Each week (or any set time period):

- Analog runs on the selected log file
- rotate and compress logfile
- rotate Analog report
- e-mail report to admin (or any e-mail address)
- After 4 weeks (or any number) overwrite files with new ones

This results in the following files being created:

- combined_log.1.gz
- combined_log.2.gz
- combined_log.3.gz
- combined_log.4.gz
- combined_log.html.1
- combined_log.html.2
- combined_log.html.3
- combined_log.html.4

By this method combined_log.html has no meaning as upon creation it is immediately rotated. In this example, apache is using the 'combined' log format described in http.conf e.g.

> # The following directives define some format nicknames for use with
> # a CustomLog directive (see below).
> #
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

> <VirtualHost www.adomain.tld>
> ...
> ...
> CustomLog /home/httpd/path_to_home_dir/logs_dir/name_of_log_file combined
> </VirtualHost>

## Method

Each minute, the system crontab checks what jobs require scheduling. Scheduling is set in the etc/crontab file.

> SHELL=/bin/bash
> PATH=/sbin:/bin:/usr/sbin:/usr/bin
> MAILTO=root
> HOME=/
> # column headings - thanks Toby
> # mins, hr, date, month, day, command
> # run-parts
> # Min Hr Date Month Day Owner Command File
> 01 * * * * root run-parts /etc/cron.hourly
> 02 4 * * * root run-parts /etc/cron.daily
> 22 4 * * 0 root run-parts /etc/cron.weekly
> 42 4 1 * * root run-parts /etc/cron.monthly

and what jobs are to be run is described in for example /etc/cron.weekly. In the above example, the directory /etc/cron.weekly is checked at 04:22 every Sunday morning.

My /etc/cron.weekly directory contains a :

> logrotate
> makewhatis.cron
> slocate.cron
> tmpwatch

The important file is logrotate:

> #!/bin/sh
> #added by bc 23/5/00 to rotate apache logs
> /usr/bin/analog -G +g/home/httpd/path_to analogue_cfg file/vdomain.cfg
> /usr/sbin/logrotate /etc/logrotate.conf

The third line runs Analog for a virtual host. The last line does the rotation. logrotate.conf contains:

> # see "man logrotate" for details
> # rotate log files weekly
> weekly
> # keep 4 weeks worth of backlogs
> rotate 4
> # send errors to root
> errors your@emailaddress
> # create new (empty) log files after rotating old ones
> create
> # uncomment this if you want your log files compressed
> #compress
> # RPM packages drop log rotation information into this directory
> include /etc/logrotate.d
> # no packages own lastlog or wtmp -- we'll rotate them here
> /var/log/wtmp {
> monthly
> create 0664 root utmp
> rotate 1
> }
> /var/log/lastlog {
> monthly
> rotate 1
> }
> # system-specific logs may be configured here
> # Added by BC 22/5/00
> # rotate log file:
> /home/httpd/company-domains.net/logs/combined_log {
> ifempty
> copytruncate
> rotate 4
> weekly
> mailfirst
> #mail your@emailaddress
> errors your@emailaddress
> compress
> postrotate
> /usr/bin/killall -HUP httpd
> endscript
> }
> # rotate Analog report:
> /home/httpd/company-domains.net/logs/combined_log.html {
> ifempty
> copytruncate
> rotate 4
> weekly
> mailfirst
> mail your@emailaddress
> errors your@emailaddress
> nocompress
> }

Note the first part of this file (up to # Added by BC 22/5/00) sets default parameters. Below my comment, parameters over-ride the defaults. One caveat of the default parameters is:

> # send errors to root
> errors your@emailaddress

This does not work as etc/crontab has: MAILTO=root which over-rides that set in logrotate.conf

The part that does the rotating/compressing/e-mailing, follows the comment:

> # system-specific logs may be configured here
> # Added by BC 22/5/00

Read man logrotate for details concerning what options may be useful to you. Another caveat is that the man file appears to indicate the order of the commands is un-important. For example (RedHat 6.1):

> nocompress

for /var/log/news/* appears after endscript. However changing this to compress will not work. It must come above postrotate.

Brian Clifton
Back to index of How-To's