日志分析工具Awstats动态分析Nginx日志实战

环境：CentOS6.6  nginx通过yum安装好

工具下载：http://pan.baidu.com/s/1jH4Hngy   （含awstats,FCGI,IP解析库）

一、日志自动切割，便于Awstats分析，详看此篇文章：http://blog.csdn.net/sinat_36255444/article/details/53837463

二、配置FCGI

tar -zxf FCGI-0.74.tar.gz

cd FCGI-0.74

perl Makefile.PL

make && make install

tar -zxf FCGI-ProcManager-0.24.tar.gz

cd FCGI-ProcManager-0.24

perl Makefile.PL

make && make install

 

#创建FCGI启动脚本

vim fcgi
#!/usr/bin/perl 
use FCGI; 
#perl -MCPAN -e 'install FCGI' 
use Socket; 
use POSIX qw(setsid); 
#use Fcntl; 
require 'syscall.ph'; 
&daemonize; 
#this keeps the program alive or something after exec'ing perl scripts 
END() { } BEGIN() { } 
*CORE::GLOBAL::exit = sub { die "fakeexit\nrc=".shift()."\n"; }; 
eval q{exit}; 
if ($@) { 
        exit unless $@ =~ /^fakeexit/; 
}; 
&main; 
sub daemonize() { 
    chdir '/'                 or die "Can't chdir to /: $!"; 
    defined(my $pid = fork)   or die "Can't fork: $!"; 
    exit if $pid; 
    setsid                    or die "Can't start a new session: $!"; 
    umask 0; 
} 
sub main { 
#$socket = FCGI::OpenSocket( "127.0.0.1:8999", 10 ); 
$socket = FCGI::OpenSocket( "/etc/nginx/fastcgi_temp/perl_cgi-dispatch.sock", 10 ); 
#use UNIX sockets - user running this script must have w access to the 'nginx' folder!! 
$request = FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%req_params, $socket ); 
if ($request) { request_loop()}; 
FCGI::CloseSocket( $socket ); 
} 
sub request_loop { 
while( $request->Accept() >= 0 ) {
#processing any STDIN input from WebServer (for CGI-POST actions) 
$stdin_passthrough =''; 
$req_len = 0 + $req_params{'CONTENT_LENGTH'}; 
if (($req_params{'REQUEST_METHOD'} eq 'POST') && ($req_len != 0) ){ 
my $bytes_read = 0; 
while ($bytes_read < $req_len) { 
my $data = ''; 
my $bytes = read(STDIN, $data, ($req_len - $bytes_read)); 
last if ($bytes == 0 || !defined($bytes)); 
$stdin_passthrough .= $data; 
$bytes_read += $bytes; 
 } 
} 
#running the cgi app 
if ( (-x $req_params{SCRIPT_FILENAME}) && #can I execute this? 
(-s $req_params{SCRIPT_FILENAME}) && #Is this file empty? 
(-r $req_params{SCRIPT_FILENAME})     #can I read this file? 
){ 
pipe(CHILD_RD, PARENT_WR); 
my $pid = open(KID_TO_READ, "-|"); 
unless(defined($pid)) { 
print("Content-type: text/plain\r\n\r\n"); 
print "Error: CGI app returned no output - Executing $req_params 
{SCRIPT_FILENAME} failed !\n"; 
next; 
} 
if ($pid > 0) { 
close(CHILD_RD); 
print PARENT_WR $stdin_passthrough; 
close(PARENT_WR); 
while(my $s = <KID_TO_READ>) { print $s; } 
close KID_TO_READ; 
waitpid($pid, 0); 
} else { 
foreach $key ( keys %req_params){ 
$ENV{$key} = $req_params{$key}; 
} 
# cd to the script's local directory 
if ($req_params{SCRIPT_FILENAME} =~ /^(.*)\/[^\/]+$/) { 
chdir $1; 
} 
close(PARENT_WR); 
close(STDIN); 
#fcntl(CHILD_RD, F_DUPFD, 0); 
syscall(&SYS_dup2, fileno(CHILD_RD), 0); 
#open(STDIN, "<&CHILD_RD"); 
exec($req_params{SCRIPT_FILENAME}); 
die("exec failed"); 
          } 
       } 
else { 
print("Content-type: text/plain\r\n\r\n"); 
print "Error: No such CGI app - $req_params{SCRIPT_FILENAME} may not exist or is 
not executable by this process.\n"; 
       } 
   } 
}

#创建完成后，需要赋予FCGI执行权限，并启动

chmod 755 fcgi

./fcgi

三、Awstats的安装与配置

tar -zxf awstats-7.5.tar.gz

mv awstats-7.5 /usr/local/awstats

#修改权限，下载下来的包是非root权限的，否则稍后*.pl会无法运行：

chown -R root.root /usr/local/awstats

chmod +x /usr/local/awstats/tools/*.pl

chmod +x/usr/local/awstats/wwwroot/cgi-bin/*.pl

#接下来执行awstats/tools下的awstats_configure.pl配置向导，生成配置文件

cd /usr/local/awstats/tools

./awstats_configure.pl

此时会出现如下提示：

----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur ----- 
This tool will help you to configure AWStats to analyze statistics for 
one web server. You can try to use it to let it do all that is possible 
in AWStats setup, however following the step by step manual setup 
documentation (docs/index.html) is often a better idea. Above all if: 
- You are not an administrator user, 
- You want to analyze downloaded log files without web server, 
- You want to analyze mail or ftp log files instead of web log files, 
- You need to analyze load balanced servers log files, 
- You want to 'understand' all possible ways to use AWStats... 
Read the AWStats documentation (docs/index.html). 
-----> Running OS detected: Linux, BSD or Unix 
-----> Check for web server install 
Enter full config file path of your Web server. 
Example: /etc/httpd/httpd.conf 
Example: /usr/local/apache2/conf/httpd.conf 
Example: c:\Program files\apache group\apache\conf\httpd.conf 
Config file path ('none' to skip web server setup): 
> none      #这里让填写网页服务器的配置文件路径，因为我们用的不是apache，所以这里要填none 
Your web server config file(s) could not be found. 
You will need to setup your web server manually to declare AWStats 
script as a CGI, if you want to build reports dynamically. 
See AWStats setup documentation (file docs/index.html) 
-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf' 
  File awstats.model.conf updated. 
-----> Need to create a new config file ? 
Do you want me to build a new AWStats config/profile 
file (required if first install) [y/N] ? y         #询问是否创建一个新的配置文件，这里填y 
-----> Define config file name to create 
What is the name of your web site or profile analysis ? 
Example: www.mysite.com 
Example: demo 
Your web site, virtual server or profile name: 
> www.haha.com      #这里让填写你的网站域名，虚拟主机名或者随便一个配置名 
-----> Define config file path 
In which directory do you plan to store your config file(s) ? 
Default: /etc/awstats 
Directory path to store config file(s) (Enter for default): 
>              #这里要填写你配置文件存放路径，我们使用它默认的路径/etc/awstats，所以直接回车即可 
-----> Create config file '/etc/awstats/awstats.www.haha.com.conf' 
 Config file /etc/awstats/awstats.www.haha.com.conf created. 
-----> Add update process inside a scheduler 
Sorry, configure.pl does not support automatic add to cron yet. 
You can do it manually by adding the following command to your cron: 
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.haha.com 
Or if you have several config files and prefer having only one command: 
/usr/local/awstats/tools/awstats_updateall.pl now 
Press ENTER to continue...   #提示不能自动加入crontab定时任务，需要稍后自己添加，我们按回车继续即可 
A SIMPLE config file has been created: /etc/awstats/awstats.www.haha.com.conf 
You should have a look inside to check and change manually main parameters. 
You can then manually update your statistics for 'www.haha.com' with command: 
> perl awstats.pl -update -config=www.haha.com 
You can also build static report pages for 'www.haha.com' with command: 
> perl awstats.pl -output=pagetype -config=www.haha.com 
Press ENTER to finish... #提示配置文件创建完成和如何更新配置及建立静态报告页，这里我们回车即可结束这个配置向导

# 修改awstats配置文件vim /etc/awstats/awstats.www.haha.com.conf

LogFile="/var/log/nginx/access.log"

LogFile="/var/log/nginx/access-%YYYY-0%MM-0%DD-24.log"

DirData="/var/lib/awstats"

意思是：读取今天和昨天的日志文件，分析结果放到var/lib/awstats

#生成awstats统计信息数据库

/usr/local/awstats/tools/awstats_updateall.pl now

四、配置日志分析页面的来访ip的地址位置显示

下载文件中有三个文件qqhostinfo.pm，qqwry.pl和QQWry.Dat，将这三个文件统统都放到/usr/local/awstats/wwwroot/cgi-bin/plugins中

#修改awstats配置文件，将LoadPlugin="hostinfo"替换为LoadPlugin="qqhostinfo"即可

vim /etc/awstats/awstats.www.haha.com.conf

LoadPlugin="qqhostinfo"

五、配置nginx

vim /etc/nginx/conf.d/default.conf

添加location站点

location ~* ^/cgi-bin/.*\.pl$ {
        root            /usr/local/awstats/wwwroot;
        fastcgi_pass    unix:/etc/nginx/fastcgi_temp/perl_cgi-dispatch.sock;
        fastcgi_index   index.pl;
        fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
        include         fastcgi_params;
    }
    location ~* ^/icon/.*\.(gif|jpg|jpeg|png|bmp|swf)$ {
        root            /usr/local/awstats/wwwroot;
        expires         7d;

配置完毕之后，检查nginx语法，然后优雅重启之后，用游览器访问

http://www.haha.com/cgi-bin/awstats.pl?config=www.haha.com

六、配置Awstats自动运行，将以下命令写入crontab中

1 * * * * /usr/local/awstats/tools/awstats_updateall.pl now

部署成功！