ELK部署

一、 安装jdk 要求在1.8版本以上

解压jdk1.8,并编辑profile文件，在最后添加如下内容：

[root@localhost elk]# tar xvf jdk-8u101-linux-i586.gz
[root@localhost elk]# vim /etc/profile
JAVA_HOME=/elk/jdk1.8.0_101
JAVA_BIN=/elk/jdk1.8.0_101/bin
JRE_HOME=/elk/jdk1.8.0_101/jre
PATH=$PATH:$JAVA_BIN
CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN  JRE_HOME PATH CLASSPATH

生效profile文件

[root@localhost elk]# source /etc/profile

二、 安装logstash，elasticsearch， kibana 的tar.gz包。

分别解压三个程序的包，不需要安装，只需要解压即可。

1.配置logstash、elasticsearch、kibana 的环境变量

[root@ZYP-TEST bin]# echo "export PATH=\$PATH:/usr/local/elk/logstash/bin" > /etc/profile.d/logstash.sh
[root@ZYP-TEST bin]# echo "export PATH=\$PATH:/usr/local/elk/elasticsearch/bin" > /etc/profile.d/logstash.sh
[root@ZYP-TEST bin]# echo "export PATH=\$PATH:/usr/local/elk/kibana/bin" > /etc/profile.d/logstash.sh
[root@ZYP-TEST bin]# . /etc/profile

2. logstash常用参数

-e :指定logstash的配置信息，可以用于快速测试；
-f :指定logstash的配置文件，可以用于生产环境；

3、启动logstash

3.1 通过-e参数指定logstash的配置信息，用于快速测试，直接输出到屏幕。

logstash -e "input {stdin{}} output {stdout{}}"
my name is zhengyansheng. //手动输入后回车，等待10秒后会有返回结果
Logstash startup completed
2015-10-08T13:55:50.660Z 0.0.0.0 my name is zhengyansheng.

这种输出是直接原封不动的返回...

3.2 通过-e参数指定logstash的配置信息，用于快速测试，以json格式输出到屏幕。

logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
my name is zhengyansheng. //手动输入后回车，等待10秒后会有返回结果
Logstash startup completed
{
"message" => "my name is zhengyansheng.",
"@version" => "1",
"@timestamp" => "2015-10-08T13:57:31.851Z",
"host" => "0.0.0.0"
}

这种输出是以json格式的返回...

4、logstash以配置文件方式启动

4.1 输出信息到屏幕

vim logstash-simple.conf 
input { stdin {} }
output {
stdout { codec=> rubydebug }

}

logstash -f logstash-simple.conf //普通方式启动

Logstash startup completed

logstash -f logstash-simple.conf --verbose //开启debug模式

Pipeline started {:level=>:info}
Logstash startup completed
hello world. //手动输入hello world.
{
"message" => "hello world.",
"@version" => "1",
"@timestamp" => "2015-10-08T14:01:43.724Z",
"host" => "0.0.0.0"
}

效果同命令行配置参数一样...

4.2 logstash输出信息存储到redis数据库中

刚才我们是将信息直接显示在屏幕上了，现在我们将logstash的输出信息保存到redis数据库中。 前提是本地(10.10.128.79)有redis数据库，那么下一步我们就是安装redis数据库.

cat logstash_to_redis.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host => '192.168.1.104'
data_type => 'list'
key => 'logstash:redis'
}
}

如果提示Failed to send event to Redis，表示连接Redis失败或者没有安装，请检查...

5、 查看logstash的监听端口号

logstash -f logstash_to_redis.conf
netstat -tnlp |grep java
tcp 0 0 :::9301 :::* LISTEN 1326/java

三、 安装redis

1.安装redis

[root@ZYP-TEST ELK]# wget http://download.redis.io/releases/redis-2.8.19.tar.gz
[root@ZYP-TEST ELK]# yum install tcl -y
[root@ZYP-TEST ELK]# tar zxf redis-2.8.19.tar.gz
[root@ZYP-TEST ELK]# cd redis-2.8.19
[root@ZYP-TEST ELK]# make MALLOC=libc
[root@ZYP-TEST ELK]# make test    //这一步时间会稍久点...
[root@ZYP-TEST ELK]# make install

//脚本执行后，所有选项都以默认参数为准即可

[root@ZYP-TEST redis-2.8.19]# cd utils/
[root@ZYP-TEST utils]# ls
build-static-symbols.tcl  hyperloglog        mkrelease.sh   redis_init_script      redis-sha1.rb         whatisdoing.sh
generate-command-help.rb  install_server.sh  redis-copy.rb  redis_init_script.tpl  speed-regression.tcl

[root@ZYP-TEST utils]# ./install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] 
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] 
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] 
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6379
Config file    : /etc/redis/6379.conf
Log file       : /var/log/redis_6379.log
Data dir       : /var/lib/redis/6379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!

2. 查看redis监控端口

[root@ZYP-TEST utils]# netstat -anlptu | grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      25909/redis-server  
tcp        0      0 :::6379                     :::*                        LISTEN      25909/redis-server

3. 查看redis进程

[root@ZYP-TEST utils]# ps -ef | grep redis
root     25909     1  0 13:56 ?        00:00:00 /usr/local/bin/redis-server *:6379

4. 测试redis是否正常工作

[root@ZYP-TEST src]# redis-cli -h 10.10.128.79 -p 6379 
10.10.128.79:6379> ping
PONG
10.10.128.79:6379> set name zhaoyuepeng
OK
10.10.128.79:6379> get name
"zhaoyuepeng"
10.10.128.79:6379> quit

5. redis的动态监控

redis-cli monitor &
OK

6、logstash结合redis工作

6.1 首先确定redis服务是启动的，然后启动redis动态监控

6.2 基于入口redis启动logstash

cat logstash_to_redis.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host => '10.10.128.79'
data_type => 'list'
key => 'logstash:redis'
}
}

logstash -f logstash_to_redis.conf --verbose
Pipeline started {:level=>:info}
Logstash startup completed
dajihao linux
{
"message" => "dajihao linux",
"@version" => "1",
"@timestamp" => "2015-10-08T14:42:07.550Z",
"host" => "0.0.0.0"
}

6.3 查看redis的监控接口上的输出

./redis-cli monitor

OK
1444315328.103928 [0 192.168.1.104:56211] "rpush" "logstash:redis" "{\"message\":\"dajihao linux\",\"@version\":\"1\",\"@timestamp\":\"2015-10-08T14:42:07.550Z\",\"host\":\"0.0.0.0\"}"

如果redis的监控上也有以上信息输出，表明logstash和redis的结合是正常的。

四、配置 elasticsearch

1.创建一个普通用户，因为elasticsearch需要用普通用户启动。

[root@localhost elk]# useradd appuser
[root@localhost elk]# chown –R appuser:appuser /usr/local/elk/elasticsearch

2.用root 创建两个文件，用于保存数据和日志

[root@localhost elk]# mkdir -p /usr/local/elk/elasticsearch/new/data
[root@localhost elk]# mkdir -p /usr/local/elk/elasticsearch/new/logs

3.编辑配置文件elasticsearch

vim /usr/local/elk/elasticsearch/config/elasticsearch.yml

加入以下内容：

cluster.name: my-test
node.name: 1
path.data: /usr/local/elk/elasticsearch/new/data
path.logs: /usr/local/elk/elasticsearch/new/logs
network.host: 10.10.128.79
http.port: 9200

4.编辑系统文件 vim /etc/sysctl.conf

加入如下内容：

vm.max_map_count=655360

5.编辑系统文件 vim /etc/security/limits.conf

加入如下内容

appuser hard nofile 65536
appuser soft nofile 65536

6.编辑系统文件 vi /etc/security/limits.d/90-nproc.conf

*          soft    nproc     2048

7.切换到appuser用户，并启动elasticsearch

[root@localhost ~]# su - appuser
[appuser@localhost ~]$ nohup elasticsearch &

8.验证

[appuser@localhost elk]$ curl http://10.10.128.79:9200/_search?pretty

配置kibana

1.修改kinaba配置文件kinaba.yml

vim /usr/local/elk/kibana/config/kibana.yml
server.port: 5601
elasticsearch.url: "http://10.10.128.79:9200"
server.host: "10.10.128.79"

2.启动kibana

nohup kibana &