ELK · Logstash的使用

最新推荐文章于 2023-01-10 15:32:20 发布
最新推荐文章于 2023-01-10 15:32:20 发布
阅读量249 收藏
点赞数
分类专栏： ElasticStack 文章标签： 中间件
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qdboi/article/details/109482261
版权

目录

四、Logstash

1、简介

在这里插入图片描述

2、功能

简单来说就是输入-过滤-输出。

3、部署安装

#检查jdk环境，要求jdk1.8+ 
java -version 

#解压安装包 
tar -xvf logstash-6.2.2.tar.gz 

#第一个logstash示例 
bin/logstash -e 'input { stdin { } } output { stdout {} }'

在这里插入图片描述

4、接收filebeat输入的日志

接下来，我们将Filebeat和Logstash整合起来，读取nginx的日志。

通过filebeat采集日志输送到logstash经过过滤存储到Elasticsearch

a.安装Nginx

apt install nginx -y 
#/usr/sbin/nginx：主程序 
#/etc/nginx：存放配置文件 
#/usr/share/nginx：存放静态文件 
#/var/log/nginx：存放日志 

#nginx服务命令 
service nginx {start|stop|restart|reload|force-reload|status|configtest|rotate|upgrade} 

#通过浏览器访问页面并且查看日志 
#访问地址：http://192.168.1.7/ 
tail -f /var/log/nginx/access.log

在这里插入图片描述

b.配置Filebeat

#vim test-nginx.yml 
filebeat.inputs: 
- type: log 
  enabled: true 
  paths: 
    - /var/log/nginx/access.log 
  tags: ["log"] 
  fields: 
    from: nginx 
  fields_under_root: false 
output.logstash: 
  hosts: ["192.168.1.7:5044"] 
  
#启动 
./filebeat -e -c test-nginx.yml 
#说明：现在启动会报错，因为Logstash还没有启动

c.配置Logstash

vim test-pipeline.conf 
#输入如下内容： 
input { 
	beats { 
		port => "5044" 
	} 
}
# 这里先不启用filter
# filter { 
#
# } 
output { 
	stdout { codec => rubydebug } 
}

#启动 --config.test_and_exit 用于测试配置文件是否正确 
bin/logstash -f test-pipeline.conf --config.test_and_exit
#正式启动 --config.reload.automatic 热加载配置文件，修改配置文件后无需重新启动 
bin/logstash -f test-pipeline.conf --config.reload.automatic

测试分别启动Filebeat和Logstash，刷新nginx页面查看输出。

{ 
    "@timestamp" => 2019-01-14T12:23:37.604Z, 
    "fields" => { 
        "from" => "nginx" 
    }, 
    "source" => "/var/log/nginx/access.log", 
    "tags" => [ 
        [0] "log", 
        [1] "beats_input_codec_plain_applied" 
    ], 
    "host" => { 
        "name" => "itcast01" 
    }, 
    "beat" => { 
        "name" => "itcast01", 
        "version" => "6.2.2", 
        "hostname" => "itcast01" 
    },
    "@version" => "1", 
    "offset" => 600, 
    "message" => "192.168.1.20 - - [14/Jan/2019:20:23:35 +0800] \"GET / HTTP/1.1\" 
    304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, 
    like Gecko) Chrome/70.0.3538.67 Safari/537.36\"", 
    "input" => { 
        "type" => "log" 
    },
    "prospector" => { 
        "type" => "log" 
    } 
} 
#可以看到，已经在控制台输出了nginx的访问日志。

d.配置filter

在前面的输出中，可以看出，虽然可以拿到日志信息，但是信息格式并不友好，比如说，不能直接拿到日志中的ip地址。

# 1、自定义nginx日志格式
vim /etc/nginx/nginx.conf 
log_format  main   '$remote_addr|$time_local|$request|'
                   '$status|$body_bytes_sent|$http_referer|'
                   '$http_user_agent|$http_x_forwarded_for|'
                   '$upstream_addr|$upstream_status|$request_time|$upstream_response_time';
access_log /usr/local/nginx/logs/access.log main;

nginx -s reload 

# 2、写nginx-patterns文件
REMOTE_ADDR              ^([\d\.]+)
TIME_LOCAL               [^\s]+\s\+\d{4}
REQUEST                  [A-Z]+\s.*\sHTTP/\d\.\d
STATUS                   \d{3}
BODY_BYTES_SENT          \d+
HTTP_REFERER             [^\|]*
HTTP_USER_AGENT          [^\|]*
HTTP_X_FORWARDED_FOR     [\d.]+|-
UPSTREAM_ADDR            [^\|]*
UPSTREAM_STATUS          [\d]{3}|-
REQUEST_TIME             [0-9]*\.[0-9]+
UPSTREAM_RESPONSE_TIME   ([0-9]*\.[0-9]+|-)$

NGINX_LOGS %{REMOTE_ADDR:remote_addr}\|%{TIME_LOCAL:time_local}\|%{REQUEST:request}\|%{STATUS:status}\|%{BODY_BYTES_SENT:body_bytes_sent}\|%{HTTP_REFERER:http_referer}\|%{HTTP_USER_AGENT:http_user_agent}\|%{HTTP_X_FORWARDED_FOR:http_x_forwarded_for}\|%{UPSTREAM_ADDR:upstream_addr}\|%{UPSTREAM_STATUS:upstream_status}\|%{REQUEST_TIME:request_time}\|%{UPSTREAM_RESPONSE_TIME:upstream_response_time}

USERHOST [\d\.]+
USERNAME [a-zA-Z]+


#3、修改test-pipeline.conf文件
input {
	beats {
		port => "5044"
	}
}
filter{
	grok{
		patterns_dir => "/opt/software/elk6.2.2/logstash-6.2.2/nginx-patterns_02"
		match => {
			"message" => "%{NGINX_LOGS}"
		}
		add_tag =>["nginx_logs"]
	}
}
output {
    stdout { codec => rubydebug }
}


#4、测试
{ 
    "input" => { 
   	 "type" => "log" 
	}, 
    "http_referer" => "-", 
    "remote_addr" => "192.168.1.20", 
    "fields" => { 
    	"from" => "nginx" 
    }, 
    "host" => { 
   	 "name" => "itcast01" 
    }, 
    "offset" => 664, 
    "request" => "GET / HTTP/1.1", 
    "message" => "192.168.1.20 - - [14/Jan/2019:22:52:49 +0800] \"GET / 
    HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 
    (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36\"", 
    "bytes_sent" => "0", 
    "status" => "304", 
    "time_local" => "14/Jan/2019:22:52:49 +0800", 
    "http_user_agent" => "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 
    (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36", 
    "prospector" => { 
         "type" => "log" 
    }, 
    "@version" => "1", 
    "remote_user" => "-", 
    "@timestamp" => 2019-01-14T14:52:58.948Z, 
    "source" => "/var/log/nginx/access.log", 
    "beat" => { 
        "name" => "itcast01", 
        "hostname" => "itcast01", 
        "version" => "6.2.2" 
    }, 
    "tags" => [ 
        [0] "log", 
        [1] "beats_input_codec_plain_applied", 
        [2] "nginx_access" 
    ] 
}

e.logstash grok内置正则过滤规则

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
HOST %{HOSTNAME}
IPORHOST (?:%{HOSTNAME}|%{IP})
HOSTPORT %{IPORHOST}:%{POSINT}

# paths
PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox
# doesn't turn into %XX
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?

# Months: January, Feb, 3, 03, 12, December
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
MONTHNUM (?:0?[1-9]|1[0-2])
MONTHNUM2 (?:0[1-9]|1[0-2])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

# Days: Monday, Tue, Thu, etc...
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)

# Years?
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

# Syslog Dates: Month Day HH:MM:SS
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG (?:[\w._/%-]+)
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}

# Shortcuts
QS %{QUOTEDSTRING}

# Log formats
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

# Log Levels
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

f.发送到Elasticsearch

#vim test-pipeline.conf 
input { 
    beats { 
          port => "5044" 
    } 
}
filter { 
    grok {
        patterns_dir => "/opt/software/logstash-6.2.2/nginx-patterns" 
        match => { "message" => "%{NGINX_ACCESS}"} 
        remove_tag => [ "_grokparsefailure" ] 
        add_tag => [ "nginx_access" ] 
    } 
}
#output {
# stdout { codec => rubydebug } 
#}
output { 
    elasticsearch { 
    	hosts => [ "192.168.1.7:9200","192.168.1.7:9201","192.168.1.7:9202" ] 
    } 
}

5、接收JSON格式的日志输出

a、修改nginx的日志格式

log_format  json  '{"@timestamp":"$time_iso8601",'
                       '"@version":"1",'
                       '"client":"$remote_addr",'
                       '"url":"$uri",'
                       '"status":"$status",'
                       '"domain":"$host",'
                       '"host":"$server_addr",'
                       '"size":"$body_bytes_sent",'
                       '"responsentime":"$request_time",'
                       '"referer":"$http_referer",'
                       '"useragent":"$http_user_agent"'
                        '}';
access_log /usr/local/nginx/logs/access_json.log json;

b、修改filtbeat配置

filebeat.prospectors:
- type: log
  enabled: true
  paths:
  - /usr/local/nginx/logs/access_json.log
  fields:
    filetype: log_nginx
  fields_under_root: true
setup.template.settings:
  index.number_of_shards: 3
output.logstash:
  hosts: ["192.168.179.128:5044"]

c、修改logstash的配置

input {
	beats {
		port => "5044"
	}
}
filter {
	json {
		source => "message"         # 将message中的键值对进行拆分
		remove_field => ["message"]
	}
}
output {
	stdout { codec => rubydebug }
}

d、启动logstash、filebeat测试访问nginx

#输出如下
{
         "filetype" => "log_nginx",
             "beat" => {
         "version" => "6.2.2",
            "name" => "localhost.localdomain",
        "hostname" => "localhost.localdomain"
    },
             "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
              "url" => "/es",
           "domain" => "192.168.179.128",
        "useragent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",
       "prospector" => {
        "type" => "log"
    },
           "client" => "192.168.179.1",
             "size" => "435",
           "source" => "/usr/local/nginx/logs/access_json.log",
           "status" => "200",
           "offset" => 3380,
             "host" => "192.168.179.128",
          "referer" => "-",
         "@version" => "1",
    "responsentime" => "0.006",
       "@timestamp" => 2020-11-06T07:24:32.000Z
}

6、自定义Template模板配置

{
    "template": "nginx-json*",
    "version": 50001,
    "settings": {
        "index.refresh_interval": "5s"
    },
    "mappings": {
        "_default_": {
            "_all": {
                "enabled": true,
                "norms": false
            },
            "dynamic_templates": [
                {
                    "message_field": {
                        "path_match": "message",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "text",
                            "norms": false
                        }
                    }
                },
                {
                    "string_fields": {
                        "match": "*",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "text",
                            "norms": false,
                            "analyzer": "ik_max_word",
                            "fields": {
                                "keyword": {
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                }
            ],
            "properties": {
                "@timestamp": {
                    "type": "date",
                    "include_in_all": false
                },
                "@version": {
                    "type": "keyword",
                    "include_in_all": false
                }
            }
        }
    }
}

在logstash的配置文件中加入template配置

input {
        beats {
                port => "5044"
        }
}
filter {
  json {
    source => "message"         # 将message中的键值对进行拆分
    remove_field => ["message"]
  }
}
output {
        if [filetype] == "log_nginx"{
                elasticsearch{
                        hosts=>["192.168.179.128:9200"]
                        index => "nginx_json_%{+YYYY-MM-dd}.log"
                        template_overwrite => true
                        template => "/opt/software/elk6.2.2/logstash-6.2.2/test-nginx-dynamic-template.json"
                        template_name => "ngins-json*"
                }
        }
}
发抖吧小喵喵
关注
专栏目录
ELK——Logstash filter的使用和Kibana应用
w1206507055的博客
06-18 856
Logstash filter 的使用和Kibana应用
ELK安装--Logstash部署
qq_51085767的博客
03-08 976
Logstash部署
logstash -e ""
大JAVA解决方案
04-29 480
[wangshumin@CentOSNode1 bin]$ logstash -e ""Sending Logstash's logs to /home/wangshumin/ekl/logstash-5.3.1/logs which is now configured via log4j2.properties[2018-04-29T11:29:40,431][INFO ][logstash.p...
ELK日志分析系统&Sentil插件邮件报警
小啊宇的博客
09-27 957
ELK的具体详解上篇文章我已经说过了这里我就不说了 下面我来讲解一下Sentil插件 Sentinl插件 sentinl是一个免费的kibana监控预警与报告插件，与付费软件X-Pack功能类似，可以实现监控并发报警邮件。 对于Kibana的一些数据我们有时候是想要对某些字段进行持续关注的，这时候通过报警的手段就可以大幅提升对这些信息状态了解的及时性及可靠性。使用sentinl插件就可以帮助我们实现这个功能。 sentinl安装 下载地址：https://github.com/sirensolutions.
在logstash7.4中实现不同来源数据输出到Elasticsearch的指定索引中
wjzt7322的博客
12-18 1893
需求分析 logstash可以采集不同来源、不同格式的数据，经过清洗、转换后统一存储到Elasticsearch中。为了区别不同来源的的数据，在Elasticsearch中可以分别建立索引。为了实现这个需求，需要在Logstash定义不同的数据源，根据数据源匹配对应的输出。 需求实现 1、首先在Elasticsearch中为不同来源的数据建立映射（mapping），定义每个字段的数据类型。 2、其...
Logstash配置详解
u012017645的专栏
10-17 2万+
Logstash配置详解
logstash-template模板：logstash.json
06-14
该资源为logstash模板，在导入数据的时候，需要对数据格式进行格式化，可提前配置template模板进行覆盖，有需要可自行下载
ELK logstash 配置
最新发布
09-22
ELK（Elasticsearch, Logstash, Kibana）是一个开源的日志分析平台，其中Logstash是用于日志收集、处理和转发的工具。在Logstash中，可以使用配置文件来定义输入（input）、过滤器（filter）和输出（output）等部分...
Spring Boot 使用 logback、logstash、ELK 记录日志文件的方法
08-28
Spring Boot 使用 logback、logstash、ELK 记录日志文件的方法 Spring Boot 是一个基于 Java 的框架，用于构建可靠的、可维护的、灵活的 web 应用程序。在日志记录方面，Spring Boot 默认使用 logback 记录日志，...
ELK logstash jdk版本查询
09-09
要查询ELK中的Logstash和JDK的版本，你可以按照以下步骤进行操作： 1. 解压Logstash二进制包：使用以下命令解压logstash-7.5.0.tar.gz文件到/usr/local目录中： 2. 启动Logstash：使用以下命令在终端中查看数据...
Logstash设置默认template
等风来也chen
10-31 4203
指定一个默认的映射文件，将其命名未test.json。位于/opt/logstash/config/templates/下。内容大致如下 { "template" : "logstash", "settings" : { "index.number_of_shards" : 2, "number_of_replicas" : 1, "index.refresh_in...
Logstash简单介绍
热门推荐
迷途的攻城狮
06-22 5万+
Logstash入门介绍   一、Logstash简介
使用logstash将项目的日志存储到Elasticsearch中（详细！新手避坑点！）
m0_73761022的博客
01-10 2441
使用logstash将项目的日志存储到Elasticsearch中新手教程，以及避坑点，和详细的教程。
【ELK】FileBeat配置说明
没枕头我咋睡觉
09-03 408
Filebeat Configuration Example ############### Filebeat ############# filebeat: # List of prospectors to fetch data. prospectors: - # paths指定要监控的日志 paths: - /var/log/*.log #指定被监控的文件的编码类型使用plain和utf-8都是可以处理中文日志.
ELK配置总结
weixin_33728708的博客
09-21 152
在经过了近半个月的ELK环境的搭建配置，我做了我个人的工作总结，和大家分享一下。一、命令总结1.1、Es服务端口查看# netstat -nlpt | grep -E"9200|9300"1.2、Es插件安装和移除# ./bin/plugin install file:///home/apps/license-2.3.3.zip# ./bin/plugin install...
2020.12-elasticsearch学习-使用logstash从本地SQL类数据库中向elasticsearch导入数据
CuriousLiu的博客
12-15 777
2020.12-elasticsearch学习-使用logstash从本地SQL类数据库中向elasticsearch导入数据 Reference 如何安装elasticsearch栈中的logstash：https://blog.csdn.net/UbuntuTouch/article/details/99655350 从mariadb向elasticsearch中导入数据：https://www.cnblogs.com/mikeluwen/p/7686663.html 如何安装ela..
ELK 5.0.1+Filebeat5.0.1 for LINUX RHEL6.6 监控MongoDB日志
clm20482的博客
02-14 174
ELK5.0.1搭建用到的工具有： filebeat-5.0.1-linux-x86_64.tar.gz logstash-5.0.1.tar.gz elasticsearch-5.0.1.tar.gz kiba...
Logstash input输入 beats插件 和 syslog插件
周天祥的博客
03-18 1万+
Logstash input输入 beats插件 和 syslog插件 Logstash input多个输入插件同时使用 Logstash -7.2.0 filter使用的插件：grok、kv、urldecode、date、mutate、geoip 1、先看总体配置logstash.conf （执行时请去除所有中文注释） input { #beats输入插件 beats ...
elk-安装配置使用-进阶篇logstash高级配置
lovelyesz的博客
01-22 705
logstash上篇已经说过了是用于在elk系统中是充当一个管道的作用，用来将filebeat中的数据传输到elasticsearch中。 其实logstash还可以做一件事，设置elasticsearch的index、targer、field（索引、目标、字段），你可以做一个简单的理解索引是数据库，目标是表，字段就是字段咯，那么设置这些有什么用呢？ 1.搜索日志的时候可以用了做查询条；2.可...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值