1. 安装
基于rpm方式安装logstash:
(1)下载软件包
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.5-x86_64.rpm
(2)安装logstash
rpm -ivh logstash-7.17.5-x86_64.rpm
(3)验证logstash版本
ln -svf /usr/share/logstash/bin/logstash /usr/local/sbin
logstash -V
(4)基于命令行启动logstash实例
说明:stdout{}就是stdout { codec => rubydebug },标准输出
logstash -e "input { stdin { type => stdin } } output { stdout { } }"
(5)测试logstash
自行输入数据即可。
-------------------------------------------------------------------------
基于二进制方式安装logstash:
(1)下载软件包
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.5-linux-x86_64.tar.gz
(2)解压软件包
tar xf logstash-7.17.5-linux-x86_64.tar.gz -C /data/tools/logstash/
(3)验证logstash版本
ln -svf /data/tools/logstash/logstash-7.17.5/bin/logstash /usr/local/sbin/
logstash -V
(4)基于命令行启动logstash实例
logstash -e "input { stdin { type => stdin } } output { stdout { } }"
(5)测试logstash
自行输入数据即可。
2. 配置
2.1. input为stdin,output为codec => rubydebug的案例
https://www.elastic.co/guide/en/logstash/7.17/introduction.html
编写第一个logstash配置文件
(1)编写配置文件
cd /etc/logstash/
[root@elk ~]# cat > conf.d/01-stdin-to-stdout.conf <<EOF
input {
stdin { type => stdin }
}
output {
stdout { }
}
EOF
(2)启动logstash实例
[root@elk1 ~]# logstash -f conf.d/01-stdin-to-stdout.conf
2.2. logstash通用字段
所有input 插件 都支持的选项 | ||
Setting | Input type | Required |
添加字段 | ||
指定对事件采用何种解码方式 | ||
是否启用度量收集,用于监控 | ||
对事件编号,用以区分相同类型的事件 | ||
对事件添加标签 | ||
设置type字段的值,用于区分事件 |
所有filter 插件 都支持的选项 | ||
Setting | Input type | 含义 |
添加字段 | ||
添加标签 | ||
是否启用度量收集,用于监控 | ||
对事件编号,用以区分相同类型的事件 | ||
定期向output插件刷新(发送)事件,而不需要等待Logstash缓冲区满或者事件被明确地推送 | ||
移除字段 | ||
移除标签 |
所有output 插件 都支持的选项 | ||
Setting | Input type | 含义 |
是否启用度量收集,用于监控 | ||
对事件编号,用以区分相同类型的事件 |
2.3. input为filebeat,output为elasticsearch的案例
#前提:filebeat采集的nginx的日志是json格式的
[root@elk1~]# cat > conf.d/02-beats-to-es.conf <<EOF
input {
# 指定输入的类型是一个beats
beats {
# 指定监听的端口号
port => 8888
#用json解码输入事件
codec => json
}
}
output {
# 将数据在标准输出显示
stdout {}
# 将数据写入ES集群
elasticsearch {
# 指定ES主机地址
hosts => ["http://10.0.0.101:9200"]
# 指定索引名称
index => "logstash-nginx-json"
#连接es的用户名密码
user => "logstash-user"
password => "123456"
}
}
EOF
(2)启动logstash实例
[root@elk1 ~]# logstash -f conf.d/02-beats-to-es.conf
(3)filebeat上配置往logstash发送数据(见filebeat教程2.16章节)
2.4. input插件之kafka
logstash从kafka拉取数据并解析json格式案例
[root@elk101 ~]# cat > conf.d/08-kafka-to-stdout.conf <<EOF
input {
kafka {
# 指定kafka集群地址
bootstrap_servers => "10.0.0.101:9092,10.0.0.102:9092,10.0.0.103:9092"
# 指定消费的topic
topics => ["topic-1"]
# 指定消费者组
group_id => "group-2"
# 指定消费的偏移量,"earliest"表示从头读取数据,"latest"表示从最新的位置读取数据.
auto_offset_reset => "earliest"
}
}
filter {
json {
# 对指定字段进行json格式解析。
source => "message"
}
mutate {
remove_field => [ "agent","log","input","host","ecs","tags","@version" ]
}
}
output {
stdout {}
}
EOF
(2)启动logstash实例
[root@elk1 ~]# logstash -f conf.d/08-kafka-to-stdout.conf
2.5. filter插件之grok,geoip,date处理数据
------------------------------------
grok相关的pattern定义查看方法:
]# ls `find /usr/share/logstash/ -type d -name "legacy"`|sort|xargs -n6|column -t
aws bacula bind bro exim firewalls
grok-patterns haproxy httpd java junos linux-syslog
maven mcollective mcollective-patterns mongodb nagios postgresql
rails redis ruby squid
-------------------------------------
logstash解析filebeat发过来的nginx原生日志并分析IP地址实战
(1)logstash配置文件编写
[root@elk1 ~]# cat > conf.d/03-beats-grok_geoip-es.conf <<EOF
input {
beats {
port => 8888
}
}
filter {
#从源数据中正则匹配信息形成新字段
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => [ "agent","log","input","host","ecs","tags" ]
}
#解析ip增加经纬度信息
geoip {
source => "clientip"
}
# 用预定义的格式匹配时间字符串,若匹配则并把时间保存到target指定的字段内
date {
# "22/Nov/2015:11:57:34 +0800"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
# 匹配时区
timezone => "Asia/Shanghai"
# 将转后的日期替换为指定字段,若不指定,则默认值为"@timestamp"
target => "@timestamp"
}
}
output {
# stdout {}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logstash-nginx"
}
}
EOF
[root@elk101 ~]# logstash -rf conf.d/03-beats-grok_geoip-es.conf
2.6. filter之grok自定义正则
grok自定义正则模式:
[root@elk1 logstash]# cat diy-patterns/01-pattern
YEAR [\d]{4}
AUTHOR [A-Z]+
BOOKNUMBER [0-9]{2}
[root@elk1 logstash]# cat > conf.d/04-stdin-grok-diy-pattern-stdout.conf <<EOF
input{stdin { }}
filter{
grok {
patterns_dir => ["./diy-patterns"]
match => {"message"=> "%{YEAR:year} %{AUTHOR:author} %{BOOKNUMBER:booknumber}"}
}
}
output{ stdout{ }}
EOF
#启动logstash
logstash -rf conf.d/04-stdin-grok-diy-pattern-stdout.conf
标准输入端输入: 2014 ZMS 11
输出端输出:
{
"booknumber" => "11",
"message" => " 2014 ZMS 11",
"host" => "elk1",
"@version" => "1",
"year" => "2014",
"@timestamp" => 2024-07-28T02:55:55.306Z,
"author" => "ZMS"
}
2.7. filter之Mutate插件
Mutate 插件主要是对字段进行、类型转换、删除、替换、更新等操作,可以使用以下函数:
插件 | 用法 | 解释 |
convert | convert => { "fieldname" => "integer"} | 变更字段的值的类型 |
copy | copy => { "source_field" => "dest_field" } | 拷贝一个字段生成一个新的字段, 如果字段存在则覆盖 |
gsub | gsub => [ "fieldname", "/", "_" ] | 用字符串替换正则匹配到字符串, 仅支持字符串或字符串数组的字段。 无视其他类型数据 |
join | join => { "fieldname" => "," } | 使用分隔符或字符串连接数组。 对非数组字段不执行任何操作。 |
lowercase | lowercase => [ "fieldname" ] | 将字符串转换为其小写 |
merge | merge => { "dest_field" => "added_field" } | 把added_field添加到dest_field形成数组 |
coerce | coerce => { "field1" => "default_value" } | 为已存在但为空的字段设置默认值 |
rename | rename => { "HOSTORIP" => "client_ip" } | 如果目标字段已存在,则替换其值。 如果源字段之一不存在,则不对该字段执行任何操作。 重命名多个字段时,不能保证操作的顺序。 |
replace | replace => { "message" => "%{source_host}: new message" } | 将字段的值替换为新值,如果该字段不存在,则添加该字段。新值可以包含%{foo}字符串以帮助您从事件的其他部分构建新值。 |
strip | strip => ["field1", "field2"] | 从字段中去除空格。注意:这仅适用于前导和尾随空格。 |
update | update => { "sample" => "My new message" } | 使用新值更新现有字段。如果该字段不存在,则不会采取任何操作。 |
uppercase | uppercase => [ "fieldname" ] | 将字符串转换为其大写等效项。 |
capitalize | capitalize => [ "fieldname" ] | 将字符串转换为其大写等效项。 |
2.8. logstash的if条件判断(单分支,双分支,多分支)
[root@elk1 logstash]# cat > conf.d/05-if-stdout.conf <<EOF
input {
beats {
port => 8888
type => "beats"
tags => "linux"
}
tcp {
port => 9999
type => "tcp"
tags => "linux"
}
http {
type => "http"
tags => "windows"
}
}
filter {
#单分支判断
if "linux" in [tags] {
mutate { add_field => {"custom-type-1" => "1-if-linux"} }
}
#双分支判断
if "linux" in [tags] {
mutate { add_field => {"custom-type-2" => "2-if-linux"} }
}else {
mutate { add_field => {"custom-type-2" => "2-if-windows"} }
}
#多分支判断
if [type] == "beats" {
mutate { add_field => {"custom-type-3" => "3-if-beats"} }
} else if [type] == "tcp" {
mutate { add_field => {"custom-type-3" => "3-if-tcp"} }
} else {
mutate { add_field => {"custom-type-3" => "3-if-http"} }
}
}
output {
stdout {}
}
EOF
#启动logstash
logstash -rf conf.d/05-if-stdout.conf
#测试往tcp 9999端口输入数据
echo 1111 | nc 10.0.0.101 9999
#输出端查看
{
"port" => 34362,
"message" => "1111",
"custom-type-2" => "2-if-linux",
"@version" => "1",
"custom-type-1" => "1-if-linux",
"host" => "elk1",
"type" => "tcp",
"@timestamp" => 2024-07-28T05:07:25.842Z,
"tags" => [
[0] "linux"
],
"custom-type-3" => "3-if-tcp"
}
#测试往http 8080端口输入数据
echo -e "GET / HTTP/1.0\n" | nc 10.0.0.101 8080
#输出端查看
{
....
"tags" => [
[0] "windows"
],
"type" => "http",
"custom-type-3" => "3-if-http",
"custom-type-2" => "2-if-windows"
...
}
2.9. logstash的pipline案例
logstash的pipline案例
(1)编写配置文件
[root@elk1 logstash]# cat > conf.d/06-pipeline-beats.conf <<EOF
input {
beats {
port => 8888
type => "beats"
}
}
filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => [ "agent","log","input","host","ecs","tags" ]
}
geoip {
source => "clientip"
add_field => {"custom-type" => "beats"}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
timezone => "Asia/Shanghai"
}
}
output {
# stdout {}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pipeline-beats"
}
}
EOF
[root@elk1 logstash]# cat > conf.d/06-pipeline-http.conf <<EOF
input {
http {
type => "http"
}
}
filter {
mutate {
add_field => { "custom-type" => "http" }
}
}
output {
# stdout {}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pipeline-http"
}
}
EOF
[root@elk1 logstash]# cat > conf.d/06-pipeline-tcp.conf <<EOF
input {
tcp {
port => 9999
type => "tcp"
}
}
filter {
grok {
# 指定加载pattern匹配模式的目录,可以是相对路径,也可以是绝对路径
patterns_dir => ["./diy-patterns"]
# 基于指定字段进行匹配
match => {"message"=> "%{YEAR:year} %{AUTHOR:author} %{BOOKNUMBER:booknumber}"}
add_field => {"custom-type" => "tcp"}
}
}
output {
# stdout {}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pipeline-tcp"
}
}
EOF
(2)修改pipline的配置文件
[root@elk1 logstash]# cat > pipelines.yml <<EOF
- pipeline.id: pipeline-beats
path.config: "/etc/logstash/conf.d/06-pipeline-beats.conf"
- pipeline.id: pipeline-tcp
path.config: "/etc/logstash/conf.d/06-pipeline-tcp.conf"
- pipeline.id: pipeline-http
path.config: "/etc/logstash/conf.d/06-pipeline-http.conf"
EOF
(3)启动logstash实例
logstash --path.settings /etc/logstash
2.10. filter之useragent插件
Useragent 插件可以根据请求中的 user-agent 字段,解析出浏览器设备、操作系统等信息
[root@elk1 logstash]# cat > conf.d/07-beats-filter-useragent-to-stdout.conf <<EOF
input {
beats {
port => 8888
codec => json
}
}
filter {
useragent {
#指定解析数据的源字段
source => "http_user_agent"
#把解析的信息放到目标字段中
target => "useragent"
}
}
output {
stdout {}
}
EOF
#启动logstash
logstash -rf conf.d/07-beats-filter-useragent-to-stdout.conf
#查看结果
{
..........
"http_user_agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.95 Safari/537.36",
"useragent" => {
"os_version" => "10",
"os" => "Windows",
"device" => "Other",
"name" => "Chrome",
"os_major" => "10",
"version" => "122.0.6261.95",
"major" => "122",
"minor" => "0",
"patch" => "6261",
"os_name" => "Windows",
"os_full" => "Windows 10"
}