主机(服务端Server): elaticsearch
, kibana
, fluentd aggregator
安装在这台机器上
IP:192.168.1.5
节点机(客户端Client):
- Node1: IP:
192.168.1.6
Nginx Server - Nodel2: IP:
192.168.1.7
django Server
注意:1、2、3步服务端和客户端都要安装
1. 首先安装ruby
因为系统自带的ruby版本太低, 先删除掉, 然后安装依赖软件
安装依赖软件
yum -y install zlib-devel libcurl-devel
接着安装yaml
tar xvf /soft/yaml-0.1.4.tar.gz # 官网下载的
cd yaml-0.1.4
./configure&&make&&make install
安装ruby
tar xvf /soft/ruby-1.9.3-p484.tar.gz
cd ruby-1.9.3-p484
./configure --prefix=/usr/local/ruby --enable-shared --disable-install-doc --with-opt-dir=/usr/local/ && make &&make install
2. 安装td-agent
下载地址:
http://174.129.37.216/redhat/x86_64/td-agent-1.1.17-0.x86_64.rpm
http://174.129.37.216/redhat/x86_64/td-libyaml-0.1.4-1.x86_64.rpm
软件包放在/soft下
yum -y install /soft/td-*
3. 安装td-agent插件
删除自带的ruby源(指向国外的服务器)
/usr/lib64/fluent/ruby/bin/fluent-gem sources --remove http://rubygems.org/
添加淘宝的源
/usr/lib64/fluent/ruby/bin/fluent-gem sources --http-proxy http://172.17.17.132:235 -a http://ruby.taobao.org/
安装secure-forward插件
/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-secure-forward
4. 安装elasticsearch(安装在服务端, 需要JDK1.7)
去官网下载RPM包
rpm -ivh /soft/elasticsearch-1.3.2.noarch.rpm
启动
/etc/init.d/elasticsearch start
5. 安装kibana(安装在服务端)
去官网下载压缩包, 直接放在网站目录下(nginx和apache都可以), 里面都是静态页面
6. 配置fluentd aggregator(安装在服务端)
默认打开的是24224端口, 请确保防火墙已经开启此端口
安装fluent-plugin-elasticsearch插件
/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-elasticsearch
开始编辑td-agent配置文件: /etc/td-agent/td-agent.conf
# Listen to incoming data over SSL 这是一种安装的连接, 必须密码才能访问服务传递数据, 不写默认是24224端口去接收数据
<source>
type secure_forward
shared_key 123456 #密码设置,随便设,保持两边一致就行
self_hostname localhost
cert_auto_generate yes
</source>
# Store Data in Elasticsearch 将端口接收到的数据存入elasticsearch
<match *.**>
type copy
<store>
type elasticsearch
host localhost
port 9200
include_tag_key true
tag_key log_name
logstash_format true
flush_interval 1s
</store>
</match>
7. 抓取tornado日志
配置客户端的td-agent
# Input
<source>
type forward
</source>
# Input
<source>
type tcp
tag u.access.internal
port 5170
format /^(?<time>[^\"]*),[^ ]* - (?<status>[^ ]*) (?<method>[^ ]*) (?<request>[^ ]*) \((?<ip>[^ ]*)\) (?<u_time>[^m]*)ms (?<tour_id>[^ ]*) (?<xforwarded_for>[^ ]*)/
types u_time:float,status:integer
time_format %Y-%m-%d %H:%M:%S
</source>
# Log Forwarding
<match *.**>
type secure_forward
shared_key 123456 #密码设置
self_hostname localhost
<server>
host 192.168.1.5
</server>
# use file buffer to buffer events log on disks
buffer_type file
buffer_path /var/log/td-agent/*.buffer
# use longer flush_interval to reduce CPU usage
flush_interval 2s
</match>