1 Logstash介绍
Logstash 是开源的服务器端数据处理管道,能够同时 从多个来源采集数据、转换数据,然后将数据发送到您最喜欢的 “存储库” 中。(我们的存储库当然是 Elasticsearch。)
2、安装jdk
# yum -y install java-1.8.0
# java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
3 安装logstash
# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.tar.gz
# tar zxf logstash-6.0.0.tar.gz -C /Data/apps/
配置logstash的环境变量
# echo "export PATH=\$PATH:/Data/apps/logstash-6.0.0/bin" > /etc/profile.d/logstash.sh
# . /etc/profile
4 查看帮助
# logstash --help
5 logstash常用参数
-e :指定logstash的配置信息,可以用于快速测试;
-f :指定logstash的配置文件;可以用于生产环境;
6、启动logstash
6.1 通过-e参数指定logstash的配置信息,用于快速测试,直接输出到屏幕。--quiet:日志输出安静模式
$ logstash -e "input {stdin{}} output {stdout{}}" --quiet
6.2
$ logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
7 logstash以配置文件方式启动
$ vim logstash.conf
input { stdin {} }
output {
stdout { codec=> rubydebug }
}
$ logstash -f logstash.conf --quie
yes ,i can
{
"@version" => "1",
"host" => "wechat1-dev.bj1.xxxx.net",
"@timestamp" => 2017-11-25T10:28:38.763Z,
"message" => "yes ,i can"
}
8 更多样例,
请参考官方文档样例:https://www.elastic.co/guide/en/logstash/current/config-examples.html
8.1 样例 elasticsearch
input { stdin { } }
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
8.2 样例 access_log
input {
file {
path => "/tmp/access_log"
start_position => "beginning"
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}
8.3 写入redis
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host => '192.168.1.104'
data_type => 'list'
key => 'logstash:redis'
}
}
技术交流
CleverCode是一名架构师,技术交流,咨询问题,请加CleverCode创建的qq群(架构师俱乐部):517133582。加群和腾讯,阿里,百度,新浪等公司的架构师交流。【架构师俱乐部】宗旨:帮助你成长为架构师!