使用logstash分析Apache日志

转自：http://blog.sina.com.cn/s/blog_a84e73f70101ck5r.html

网站刚刚上线，就发现一些扫描信息，因此，有必要搭一个日志分析系统，对web访问日 志进行分析，或许能够发现一些有意思的东西。logstash是一个非常灵活的日志分析引擎， 使用强大的elasticsearch作为数据存储、索引的数据库。

118.26.224.17 - - [10/Jun/2012:18:26:46 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 490 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 475 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:49 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"

创建日志分析环境

#!/bin/bash
# 创建日志环境的目录
mkdir loganalysis
cd loganalysis

# 下载logstash，目前是1.1.0版本
mkdir bin
cd bin
wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar

# 创建启动脚本
cat << EOF >>run.sh
#!/bin/bash
java -jar $LOGSTASH_HOME/bin/logstash-1.1.0-monolithic.jar agent -f $LOGSTASH_HOME/conf/apache-parse.conf -- web --backend elasticsearch:///?local
EOF

# 环境变量设置
cd ..
cat << EOF >>env.sh
#!/bin/bash
export LOGSTASH_HOME=`pwd`
EOF

# 下载Apache日志解析配置文件
mkdir conf
cd conf
wget http://logstash.net/docs/1.1.0/tutorials/10-minute-walkthrough/apache-parse.conf

修改apache-parse.conf

apache-parse.conf默认不使用elasticsearch，所以需要加上 elasticsearch { embedded => true }

input {
  tcp { 
    type => "apache"
    port => 3333
  } 
}

filter {
  grok {
    type => "apache"
    # See the following URL for a complete list of named patterns
    # logstash/grok ships with by default:
    # https://github.com/logstash/logstash/tree/master/patterns
    #
    # The grok filter will use the below pattern and on successful match use
    # any captured values as new fields in the event.
    pattern => "%{COMBINEDAPACHELOG}"
  }

  date {
    type => "apache"
    # Try to pull the timestamp from the 'timestamp' field (parsed above with
    # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
    timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
  }
}

output {
  # Use stdout in debug mode again to see what logstash makes of the event.
  stdout {
    debug => true
  }
  elasticsearch { embedded => true }
}

安装grok

apache-parse.conf使用grok作为输入的接口，但是grok与ubuntu主流版本(只测试过 11.10)并不兼容，按照INSTALL说明的安装无法成功。幸好，我们只要libgrok.so即可。

cd /tmp
git clone https://github.com/jordansissel/grok.git
cd grok
make libgrok.so
sudo cp libgrok.so /usr/local/lib
sudo ldconfig -v

运行logstash

$ . env.sh   #导入环境变量
$ bin/run.sh

启动完成之后，将apache的访问日志导入logstash：

nc localhost 3333 

最后使用浏览器打开http://localhost:9292/ 访问logstash的web界面。