Apache-日志分割技术及Awstats日志分析系统

前言

​ 随着网站的访问量越来越大，默认情况下 Apache 服务器产生的单个日志文件也会越来越大，如果不对日志进行分割，那么如果日志文件占用磁盘空间很大的话势必会将整个日志文件删除，这样也丢失了很多对网站比较宝贵的信息，而这些日志可以用来进行访问分析、网络安全监察、网络运行状况监控等。
​ 另外，如果服务器遇到故障时，运维人员要打开日志文件进行分析，打开的过程会消耗很长时间，也势必会增加处理故障的时间。因此管理好这些海量的日志对网站的意义很大，我们会将 Apache 的日志进行按每天的日期自动分割。

​ 下面介绍两种方法均可实现

1. Apache自带rotatelogs分割工具
2. 第三方工具cronolog分割

一 、自带rotatelogs分割工具

1.1 相关参数

• 配置网站的日志文件转交给rotatelogs分割处理
• 配置格式为

rotatelogs [ -l ] logfile [ rotationtime [ offset ]] | [ filesizeM ]

-l
使用本地时间代替GMT时间作为时间基准。

logfile
它加上基准名就是日志文件名。如果logfile中包含”%”，则它会被视为用于strftime()的格式字符串；否则它会被自动加上以秒为单位的”.nnnnnnnnnn”后缀。这两种格式都表示新的日志开始使用的时间。

rotationtime
日志文件滚动的以秒为单位的间隔时间。

offset
相对于UTC的时差的分钟数。如果省略，则假定为”0″并使用UTC时间。如果要使用北京时间，需要设定为480，因为北京时间=UTC+8小时，480秒

filesizeM
指定以filesizeM文件大小滚动，而不是按照时间或时差滚动。

1.2 配置分割

[root@localhost ~]# vi /etc/httpd/conf/httpd.conf 

# 添加2行
CustomLog "| /usr/sbin/rotatelogs -l logs/apache_access_%y%m%d.log 86400"  combined
ErrorLog "| /usr/sbin/rotatelogs -l logs/apache_error_%y%m%d.log 86400"

[root@localhost ~]# systemctl start httpd
[root@localhost ~]# cd /etc/httpd/logs/
[root@localhost logs]# date
Tue Nov 24 19:12:17 CST 2020
[root@localhost logs]# ll
total 12
-rw-r--r-- 1 root root  81 Nov 24 19:12 access_log
-rw-r--r-- 1 root root  81 Nov 24 19:12 apache_access_201124.log
-rw-r--r-- 1 root root 916 Nov 24 19:12 apache_error_201124.log

二、第三方工具cronolog

cronolog作为日志过滤程序，可用来切割linux日志文件，通过对输入的日志按文件名模板和当前日期重新编排，来按格式生成所需日志。cronolog 旨在和一个Web服务器一起使用，如Apache、Nginx

2.1 配置格式

CustomLog "| cronolog命令的绝对路径  日志文件路径/网站名%Y%m%d.log " combined
ErrorLog "| cronolog 命令的绝对路径  日志文件路径/网站名-error %Y%m%d.log"

2.2 配置分割

[root@localhost ~]# tar zxf cronolog-1.6.2.tar.gz -C /opt
[root@localhost ~]# cd /opt/cronolog-1.6.2/
[root@localhost cronolog-1.6.2]# ./configure
[root@localhost cronolog-1.6.2]# make && make install
[root@localhost ~]# which cronolog
/usr/local/sbin/cronolog
[root@localhost ~]# vi /etc/httpd/conf/httpd.conf 

# 添加2行
CustomLog "| /usr/local/sbin/cronolog logs/apache_cron_access_%y%m%d.log" combined
ErrorLog "| /usr/local/sbin/cronolog logs/apache_cron_error_%y%m%d.log"

[root@localhost ~]# systemctl restart httpd           
[root@localhost ~]# cd /etc/httpd/logs/
[root@localhost logs]# date
Tue Nov 24 19:29:51 CST 2020
[root@localhost logs]# ll
total 20
-rw-r--r-- 1 root root  162 Nov 24 19:28 access_log
-rw-r--r-- 1 root root  162 Nov 24 19:28 apache_access_201124.log
-rw-r--r-- 1 root root   81 Nov 24 19:28 apache_cron_access_201124.log
-rw-r--r-- 1 root root 1704 Nov 24 19:29 apache_cron_error_201124.log
-rw-r--r-- 1 root root 1033 Nov 24 19:27 apache_error_201124.log

三、部署AWStats日志分析系

# 安装AWStats
[root@localhost ~]# tar zxf awstats-7.7.tar.gz -C /opt/
[root@localhost ~]# cd /opt/awstats-7.7/
[root@localhost awstats-7.7]# cd
[root@localhost ~]# mv /opt/awstats-7.7/ /usr/local/awstats

# 为要统计的站点建立配置文件
[root@localhost ~]# cd /usr/local/awstats/tools/
[root@localhost tools]# chmod +x awstats_configure.pl 
[root@localhost tools]# ./awstats_configure.pl 

----- AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/httpd/conf/httpd.conf

-----> Check and complete web server config file '/etc/httpd/conf/httpd.conf'
  Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'
  Add 'Alias /awstatscss "/usr/local/awstats/wwwroot/css/"'
  Add 'Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"'
  Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'
  Add '<Directory>' directive
  AWStats directives added to Apache config file.

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.test.com	# 指定要统计的目标网站名称

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> 
   

-----> Create config file '/etc/awstats/awstats.www.test.com.conf'
 Config file /etc/awstats/awstats.www.test.com.conf created.

-----> Restart Web server with '/sbin/service httpd restart'
Redirecting to /bin/systemctl restart httpd.service

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.test.com
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue... 


A SIMPLE config file has been created: /etc/awstats/awstats.www.test.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.test.com' with command:
> perl awstats.pl -update -config=www.test.com
You can also read your statistics for 'www.test.com' with URL:
> http://localhost/awstats/awstats.pl?config=www.test.com

Press ENTER to finish...

# 修改配置文件
[root@localhost ~]# vi /etc/httpd/conf/httpd.conf 
<Directory "/usr/local/awstats/wwwroot">
    Options None
    AllowOverride None
    #Order allow,deny			# 注释
    #Allow from all				# 注释
    Require all granted			# 添加
</Directory>

# 修改站点统计配置文件，重启服务
[root@localhost ~]# systemctl restart httpd
[root@localhost ~]# vi /etc/awstats/awstats.www.test.com.conf 
LogFile="/etc/httpd/logs/access_log"		# 与日志文件路径相同
DirData="/var/lib/awstats"

# 创建指定的目录
[root@localhost ~]# mkdir /var/lib/awstats

# 使用updateall.pl更新数据
[root@localhost ~]# /usr/local/awstats/tools/awstats_updateall.pl now

优化方案

• 手动刷新比较麻烦，需要优化做个计划性任务，每5分钟自动刷新一次

[root@localhost ~]# crontab -e

*/5 * * * * /usr/local/awstats/tools/awstats_updateall.pl now

[root@localhost ~]# systemctl enable crond
[root@localhost ~]# systemctl start crond

• 优化访问路径

[root@localhost html]# vi /var/www/html/aws.html
<html>
 <head>
  <meta http-equiv=refresh content="0;url=http://www.test.com/awstats/awstats.pl?config=www.test.com">
 </head>
 <body></body>
</html>

此时，访问http://www.test.com/aws.html可直接跳转