Logstash安装配置
1.安装
从官网下载.tar.gz安装包,我这里下载的是6.4.3版本。
https://www.elastic.co/cn/downloads/logstash
解压:
tar -zxvf logstash-6.4.3.tar.gz
2.配置
logstash可以修改启动占用的JVM内存,在config目录下的jvm.options,内容如下:
当然不修改也可以。
创建你想要的conf文件,例如收集Nginx日志,输出到elasticsearch上,需要这样一个配置文件:
input {
file {
type => "flow" #定义一个类型,自己随意定义,在output模块用到
path => "/usr/local/nginx/logs/access.log" #nginx日志的位置
codec => "json" #日志过来时格式化类型
start_position => "beginning"
}
}
filter { #定义的日志过滤规则
grok {
match => { "message" => "%{COMBINEDAPACHELOG} %{QS:http_x_forwarded_for}"}
}
mutate {convert => ["upstream_time", "float"]}
}
output {
stdout {
codec => rubydebug #在终端打印出日志,便于观察和调试
}
if [type] == "flow" { #这里的if条件里的type,就是input里配置的
elasticsearch {
index => "flow-%{+YYYY.MM.dd}" #存入elasticsearch时,创建的索引名
hosts => ["172.16.185.31:9200"] #elasticsearch地址
}
}else{
elasticsearch {
index => "flowA-%{+YYYY.MM.dd}"
hosts => ["172.16.185.31:9200"]
}
}
}
这个配置文件稍微有些复杂,里面已经加了注释。哪里不懂可以单独去搜索一下。
其中input模块是你的输入源,也就是nginx的日志,output模块是输出源,这里是输出到elasticsearch,需要注意hosts的地址不要写错。filter是比较重要的模块,也是最复杂的,在这里可以定义一系列的日志过滤规则,这个可以去单独的搜索学习下logstash的fileter模块配置。
再提供一个读取nginx日志,存入到kafka的conf文件:
input {
file {
type => "flow"
path => "/usr/local/nginx/logs/access.log"
codec => "json"
start_position => "beginning"
}
}
output {
if [type] == "flow" {
kafka {
bootstrap_servers => "172.16.185.31:9092"
topic_id => 'nginx-access-kafkaceshi'
codec => "json"
}
}
}
还有利用filebeat收集日志,logstash读取后存入kafka的conf文件:
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug
}
kafka {
bootstrap_servers => "172.16.185.31:9092"
topic_id => 'nginx-kafka'
codec => "json"
}
}
3.启动
logstash启动命令:
./logstash -f /usr/local/logstash/logstash-6.4.3/config/flow-es.conf
然后日志信息就会出现在终端上!
4.错误
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>“LogStash::ConfigurationError”, :message=>“Expected one of #, input, filter, output at line 1, column 1 (byte 1) after “, :backtrace=>[”/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/compiler.rb:49:incompile_graph’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources’”, “org/jruby/RubyArray.java:2486:inmap’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/compiler.rb:10:in compile_sources’”, “org/logstash/execution/AbstractPipelineExt.java:149:ininitialize’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline.rb:22:in initialize’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline.rb:90:ininitialize’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute’”, “/usr/local/logstash/logstash-6.4.3/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state’”]}
2019-09-16T15:40:56,894[logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
此错误是配置文件出错了,仔细检查配置文件的格式!