(一)安装logstash
http://blog.csdn.net/napoay/article/details/53276758
$ tar -zxvf logstash-2.4.0.tar.gz
启动logstash:
$ cd logstash-2.4.0
$ ./bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 4
Pipeline main started
这样会以默认形式输出日志,是最简单的日志格式,输入一个单词测试:
hello world
2016-11-21T20:51:22.252Z yaopan.local hello world
设置为格式化输出:
$ ./bin/logstash -e 'input { stdin { } } output { stdout {codec => rubydebug} }'
Settings: Default pipeline workers: 4
Pipeline main started
hello
{
"message" => "hello",
"@version" => "1",
"@timestamp" => "2016-11-21T20:52:11.973Z",
"host" => "yaopan.local"
}
(二)安装 elasticsearch
解压后,安装head插件:https://www.cnblogs.com/xing901022/p/5469338.html
Head插件——学习Elasticsearch的锋刃利器!
安装
在网络畅通的情况下(需要能访问github),可以直接通过plugin进行安装。即在es的bin目录下运行:
F:\software\elasticsearch-2.2.1\bin>plugin.bat install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ....................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
.............DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksum
s if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .
md5 file to verify)
Installed head into F:\software\elasticsearch-2.2.1\plugins\head
安装完,就发现plugins下面多了一个head的文件夹。
通过上面的安装输出消息,也可以发现head插件其实就是下载了zip包,然后解压到了head目录中。因此,(也可以直接去github上面下载zip包)
开启远程访问权限:http://blog.csdn.net/u012599988/article/details/51767183
elasticsearch 外网访问9200端口访问
系统centos6.5
可以访问127.0.0.1:9200,但不能访问 公网IP:9200
后面ip就是127.0.0.1的局域网ip,如何解决?
启动:
./elasticsearch -Des.insecure.allow.root=true
然后访问:ip:9200/_plugin/head就可以访问了。
(三)logstash时区
http://blog.51cto.com/11067470/1729872
logstash @timestamp时间时区的问题
在filter里面加上这段代码即可修改。
ruby {
code => "event.timestamp.time.localtime"
}
(四)结合java log4j
http://blog.csdn.net/napoay/article/details/59810063
ELK日志处理之使用logstash收集log4J日志
java
log4j.logger.com=INFO,error,info,logstash
前两个是输出到文件,logstash是走logstash通道
#输出日志到logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.RemoteHost=139.1xx.x.xxx
log4j.appender.logstash.port=4560
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true
logstash
input {
stdin { }
log4j {
host => "139.1xx.x.xxx"
port => 4560
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch{
hosts => ["localhost:9200"]
index => "log4j-%{+YYYY.MM.dd}"
document_type => "log4j_type"
}
}
filter {
ruby {
code => "event.timestamp.time.localtime"
}
}
主意:java中的remotehost与logstash input 中的host 必须文本一致
启动:
./bin/logstash -f conf/log4j-es.conf,远程访问时防火墙放开 4560端口
这种方式同样适用于同一台物理机上的不同tomcat实例集群:
(五)其他模式
四中的模式为logstash server模式,logstash还有client模式
https://www.cnblogs.com/xing901022/p/4830684.html
[logstash-input-log4j]插件使用详解
Server模式
server模式就是把logstash作为服务器,输出日志消息的java程序所在的主机作为客户机,大致类似如下:
Logstash的插件配置如下:
input{
log4j {
mode => "server"
host => "localhost"#注意这里,这里是Logstash服务器的地址或者主机名
port => 4560
}
}
output{
stdout{}
}
java程序log4j日志配置文件如下:
<?xml version="1.0" encoding= "UTF-8" ?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" > <appender name="ConsoleAppender" class="org.apache.log4j.ConsoleAppender" > <layout class="org.apache.log4j.PatternLayout" > <param name="ConversionPattern" value="%d{yyyy/MM/dd-HH:mm:ss} >> %5p >> %t >> %l >> %m%n" /> </layout> </appender> <appender name="socketAppender" class="org.apache.log4j.net.SocketAppender"> <param name="remoteHost" value="localhost" /><!-- 远程主机地址 --> <param name="port" value="4560" /><!-- 远程主机端口 --> <param name="Threshold" value="DEBUG" /> <param name="ReconnectionDelay" value="60000" /> <param name="LocationInfo" value="true" /> </appender> <root> <priority value="debug" /> <appender-ref ref="ConsoleAppender" /> <appender-ref ref="socketAppender" /> </root> </log4j:configuration>
另外需要注意的是,如果使用server模式,监听的ip地址只能是本机地址,否则无法绑定socket。
例如,我本身的服务器地址是10.4.5.6,那么我要绑定一个远端机器,10.4.5.7,就会报如下错误:
Client模式
client模式就是把Logstash当做客户端,去请求返回java程序所在的主机输出的日志,大致如下:
logstash的配置如下:
input{
log4j {
mode => "client"
host => "10.4.5.6"
port => 9999
}
}
output{
stdout{}
}
java程序这端的log4j配置文件如下:
<?xml version="1.0" encoding= "UTF-8" ?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration xmlns:log4j=" http://jakarta.apache.org/log4j/" > <appender name="ConsoleAppender" class="org.apache.log4j.ConsoleAppender" > <layout class="org.apache.log4j.PatternLayout" > <param name="ConversionPattern" value="%d{yyyy/MM/dd-HH:mm:ss} >> %5p >> %t >> %l >> %m%n" /> </layout> </appender> <appender name="sockethubAppender" class="org.apache.log4j.net.SocketHubAppender"> <param name="port" value="9999" /> <param name="Threshold" value="INFO" /> <param name="LocationInfo" value="true" /> </appender> <root> <priority value="info" /> <appender-ref ref="ConsoleAppender" /> <appender-ref ref="sockethubAppender" /> </root> </log4j:configuration>