利用td-agent(fluentd)和elasticsearch搭建kibana可视化日志分析平台

主机(服务端Server): elaticsearch, kibana, fluentd aggregator 安装在这台机器上

IP:192.168.1.5

节点机(客户端Client):

• Node1: IP: 192.168.1.6 Nginx Server
• Nodel2: IP: 192.168.1.7 django Server

注意：1、2、3步服务端和客户端都要安装

1. 首先安装ruby

因为系统自带的ruby版本太低, 先删除掉, 然后安装依赖软件
安装依赖软件
yum -y install zlib-devel libcurl-devel
接着安装yaml

tar xvf /soft/yaml-0.1.4.tar.gz   # 官网下载的
cd yaml-0.1.4
./configure&&make&&make install

安装ruby

tar xvf /soft/ruby-1.9.3-p484.tar.gz
cd ruby-1.9.3-p484
./configure --prefix=/usr/local/ruby --enable-shared --disable-install-doc --with-opt-dir=/usr/local/ && make &&make install

2. 安装td-agent

下载地址:

http://174.129.37.216/redhat/x86_64/td-agent-1.1.17-0.x86_64.rpm

http://174.129.37.216/redhat/x86_64/td-libyaml-0.1.4-1.x86_64.rpm

软件包放在/soft下
yum -y install /soft/td-*

3. 安装td-agent插件

删除自带的ruby源(指向国外的服务器)
/usr/lib64/fluent/ruby/bin/fluent-gem sources --remove http://rubygems.org/

添加淘宝的源

/usr/lib64/fluent/ruby/bin/fluent-gem sources --http-proxy http://172.17.17.132:235 -a http://ruby.taobao.org/

安装secure-forward插件

/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-secure-forward

4. 安装elasticsearch(安装在服务端, 需要JDK1.7)

去官网下载RPM包

rpm -ivh /soft/elasticsearch-1.3.2.noarch.rpm

启动

/etc/init.d/elasticsearch start

5. 安装kibana(安装在服务端)

去官网下载压缩包, 直接放在网站目录下(nginx和apache都可以), 里面都是静态页面

6. 配置fluentd aggregator(安装在服务端)

默认打开的是24224端口, 请确保防火墙已经开启此端口
安装fluent-plugin-elasticsearch插件

/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-elasticsearch

开始编辑td-agent配置文件: /etc/td-agent/td-agent.conf

# Listen to incoming data over SSL 这是一种安装的连接, 必须密码才能访问服务传递数据, 不写默认是24224端口去接收数据
<source>
  type secure_forward
  shared_key 123456 #密码设置，随便设，保持两边一致就行
  self_hostname localhost
  cert_auto_generate yes
</source>

# Store Data in Elasticsearch   将端口接收到的数据存入elasticsearch
<match *.**>
  type copy
  <store>
    type elasticsearch
    host localhost  
    port 9200
    include_tag_key true
    tag_key log_name
    logstash_format true
    flush_interval 1s
  </store>
</match>

7. 抓取tornado日志

配置客户端的td-agent

# Input
<source>
  type forward
</source>

# Input
<source>
  type tcp
  tag u.access.internal
  port 5170
  format /^(?<time>[^\"]*),[^ ]* - (?<status>[^ ]*) (?<method>[^ ]*) (?<request>[^ ]*) \((?<ip>[^ ]*)\) (?<u_time>[^m]*)ms (?<tour_id>[^ ]*) (?<xforwarded_for>[^ ]*)/
  types u_time:float,status:integer
  time_format %Y-%m-%d %H:%M:%S
</source>

# Log Forwarding
<match *.**>
  type secure_forward
  shared_key 123456 #密码设置
  self_hostname localhost
  <server>
    host 192.168.1.5
  </server>

  # use file buffer to buffer events log on disks
  buffer_type file
  buffer_path /var/log/td-agent/*.buffer

  # use longer flush_interval to reduce CPU usage
  flush_interval 2s
</match>