Awstats with virtual hosts on Apache

Awstats is a very good weblog analyzer. It produce loads of statistics about your visitors and their behaviour. The first time it can be a bit tricky to setup if you have many virtual hosts. However good the application is, the documentation is a classical example of when programmers write the manual. They understand everything so it all makes perfect sense to them. This article describes how to setup the awstats package with authentication for multiple virtual hosts and how to use logrotate and scripts to automate tasks.

Installing awstats
For this I used Fedora Core 2 with Apache 1.3 installed in /opt/apache-1.3/. I install awstats from an RPM downloaded from rpmfind.net. At the time of my writing this the latest rpm available is awstats-6.2-1.1.fc2.rf.noarch.rpm. The default installation directory is /var/www/awstats and the config files are placed in /etc/awstats. The installation directory is not the optimal as didn't install apache in that location but it doesn't really matter. After installation these lines should be present at the bottom of your httpd.conf files. If commented out, uncomment them.

Alias /awstatsclasses "/var/www/awstats/classes/"Alias /awstatscss "/var/www/awstats/css/"Alias /awstatsicons "/var/www/awstats/icon/"ScriptAlias /awstats "/var/www/awstats/"<Directory "/var/www/awstats">    Options None    AllowOverride None    Order allow,deny    Allow from all</Directory>The way I set it up requires basic authentication. Adding the required lines to the Directory part leaves it like below. In this case I specified the password file to reside in the same place as the rest of the awstats configuration. The awstats.pwd file will be created later.

<Directory "/var/www/awstats">    Options None    AllowOverride None    Order allow,deny    Allow from all    AuthType Basic    AuthName stats    AuthUserFile /etc/awstats/awstats.pwd    require valid-user</Directory>Setup apache logging
Awstats recommend you to split your logs between your virtual hosts so that each virtual host has its own apache access_log file. Even so I like to keep logging for all the virtual hosts in the same log file. I think it is a bit easier to handle the logfile and it also simplifies logrotating.

As I'm keeping all virtual hosts in the same logfile I need to make sure the virtual host is written to each line. This does not come out of the box and a small change in httpd.conf is needed. In httpd.conf you will find a few lines that specify different log formats. Per default the combined log format should be used. This is also what awstats recommend. It is defined in httpd.conf.

LogFormat "%h %l %u %t /"%r/" %>s %b /"%{Referer}i/" /"%{User-Agent}i/"" combinedAs I want to be able to differantiate log entries between different virtual hosts I want the combined format plus the virtual host. So I create my own log format that I call cplus. This format has %v prepended to it which will put the virtual host first on each log entry. I add the following line to httpd.conf

LogFormat "%v %h %l %u %t /"%r/" %>s %b /"%{Referer}i/" /"%{User-Agent}i/"" cplusI also need to tell apache to use the cplus log format instead of the combined format by changing the CustomLog directive to look like below.

CustomLog /opt/apache-1.3/logs/access_log cplusI restart apache and have a look in access_log to verify that the virtual host is written on each log entry.

Configure awstats
With apache done it is time to configure awstats. I'm only interested in the file awstats.model.conf in /etc/awstats. So the first thing I do is to remove the everything else from /etc/awstats that was created during installation. This file, awstats.model.conf, is going to be the template for all other awstats.*.conf files as awstats needs one config file per virtual host. Setting up a new virtual host manually is boring as well as error prone so I create scripts to do most of the job. This is also the case with awstats.

There are several configuration changes needed in awstats.model.conf. As I don't use the combined log format I have to specify a custom log format. In several places I change the values to something that is easier to parse with a script. Applicable entries and their values are:

LogFile="/opt/apache-1.3/logs/access_log"LogFormat = "%virtualname %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"SiteDomain="$DOMAIN"HostAliases="$ALIASES"DirData="/home/$USERNAME/awstats"AllowAccessFromWebToAuthenticatedUsersOnly=1AllowAccessFromWebToFollowingAuthenticatedUsers="$USERNAME"There are loads of other stuff you can tweak but these are the ones important for this setup.

Set up statistics for a host
<VirtualHost *:80>    ServerName www.mysite300.org    ServerAlias mysite300.org    DocumentRoot /home/mysite300_org/www    ServerAdmin info@mysite300.org</VirtualHost>Everything is now in place to setup awstats for an individual virtual host. The above is a minimal example of a virtual host section in httpd.conf. If I want to manually setup awstats for www.mysite300.org I have to create a directory to host the statistics files. I have to copy awstats.model.conf to awstats.www.mysite300.org and change all the configuration values by hand. Then I need to run htpasswd to add a username and password to /etc/awstats/awstats.pwd to allow someone to see the statistics.

As I am lazy I want to cut this three minute job to a few seconds. I have created the following small script, addstat.sh, to ask the right questions and then perform the task. This script is run as root. It could be prettier but it gets the job done.

#!/bin/bashecho "Enter the username:"read WUSERNAMEACCESS_FILE="/etc/awstats/awstats.pwd"STAT_DIR="/home/$WUSERNAME/awstats"echo "Enter the password:"read PASSWORDecho "Enter the main domain:"read DOMAINecho "Enter aliases separated by space:"read ALIASES# Create the statistics directoryif [ -d $STAT_DIR ]; then    echo "Statistics dir already exist"else    mkdir $STAT_DIRfi# Create the virtual host awstats.confcat /etc/awstats/awstats.model.conf | /sed -e "s$DOMAIN/$DOMAIN/g" | /sed -e "s$USERNAME/$WUSERNAME/g" | /sed -e "s$ALIASES/$ALIASES/g" > /"/etc/awstats/awstats.$DOMAIN.conf"# Add user/password to password fileif [ -e $ACCESS_FILE ]; then    /opt/apache-1.3/bin/htpasswd -bm $ACCESS_FILE $WUSERNAME $PASSWORDelse    /opt/apache-1.3/bin/htpasswd -bm -c $ACCESS_FILE $WUSERNAME $PASSWORDfiAfter running this script to setup awstats for the example virtual host the user mysite300_org can go to http://www.mysite300.org/awstats/awstats.pl to view his statistics. The user will be presented with a login form and only allowed through if he presents the correct credentials.

Generating statistics
Now that I have run the script and setup statistics for this virtual host I want to generate statistics as well. Because it is still empty. I can do this manually through

/var/www/awstats/awstats.pl -update -config=www.mysite300.orgI have many virtual hosts and I want the statistics to be updated every day. So this is not an option. Creating a script and run it from a cron job could be a good idea. But an even better idea is to use logrotate to rotate the apache logs and have logrotate run our script.

A simple script, genstat.sh, to generate statistics for all awstats.*.conf files in /etc/awstats except awstats.model.conf is needed. I put the script in /root/scripts.

#!/bin/bashCONF_DIR="/etc/awstats"CONF_EXT="conf"if [ ! -d "$CONF_DIR" ]; then  echo "$CONF_DIR does not exist"  exit 1else  cd $CONF_DIRfifor cfg in *.$CONF_EXT; doif [ "$cfg" != "awstats.model.conf" ]; then   site=`echo $cfg|sed -e "s/awstats.//g" | sed -e "s/.conf//g"`   /var/www/awstats/awstats.pl -update -config="$site"fidoneThen I need to configure logrotate.conf in /etc to run this script prior to rotating the logs. If rpm was used to install apache it could have its own file in /etc/logrotate.d/. I normally install apache manually, so I add this entry directly in logrotate.conf.

/opt/apache-1.3/logs/access_log {   rotate 31   daily   compress   sharedscripts   prerotate     /root/scripts/dailystat.sh   endscript   postrotate     /opt/apache-1.3/bin/apachectl restart   endscript}There is one problem one can encounter with logrotate. There are many vulnerable applications on the web. When hosting applications I don't have any control over I (ofcourse) try to secure the system as good as possible. As a security measure I have /tmp mounted with the options nosuid and noexec. The specific line in /etc/fstab looks like below.

LABEL=/tmp /tmp ext3 defaults,nosuid,noexec 1 2Crackers and worms often exploit vulnerable web applications to upload scripts and run them from /tmp. Mounting /tmp with the noexec option is a good way to make sure this cannot happen even if there are vulnerable applications on the system. If you haven't done so already - now is a good time to fix this. However, this also stops logrotate as it runs scripts from /tmp (or whatever you have TMPDIR defined as).

Logrotate is run daily per default. The acutal script can be found in /etc/cron.daily/logrotate. To fix this I create a tmp directory, /tmp_safe, that is going to be used by logrotate only. Then I add the following to the top of /etc/cron.daily/logrotate.

export TMPDIR=/tmp_safeNo logrotate will use /tmp_safe as a tmp directory instead of /tmp. All other applications running on the system will still use /tmp as normal.

Wrapping it up
Now I have a working awstats installation and logrotate rotates my apache access_log. The logs are compressed and saved for 31 days. When I want to add statistics possibilites for a virtual host I just run the script addstat.sh. Life is simple again.

Files created

/root/scripts/addstat.sh
/root/scripts/genstat.sh
Files changed

/opt/apache-1.3/conf/httpd.conf
/etc/awstats/awstats.model.conf
/etc/logrotate.conf
/etc/cron.daily/logrotate
/etc/fstab