ELK系列（四） Logstash8.9.2搭建

已于 2024-01-16 16:57:29 修改
阅读量612 收藏 13
点赞数 8
分类专栏： ELK 文章标签： elk
于 2024-01-16 16:52:46 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/BourneSu/article/details/135504094
版权

ELK系列传送门

ELK系列（一） ElasticSearch 8.9.2集群搭建

ELK系列（二）Kafka集群3.4.0搭建

ELK系列（三） Kibana8.9.2搭建

ELK系列（四） Logstash8.9.2搭建

ELK数据流规划

ELK接入nginx-acces日志数据流规划


数据流向
		filebeat -------> kafka ----> logstash ----> ES --------> kibana
		 收集              mq          处理日志     存储、搜索     展示、分析
                                                       |
										               |
								                   elastalert2---钉钉告警


ES集群搭建已完成 可见传送门ELK系列（一） ElasticSearch 8.9.2集群搭建
Kafka集群搭建已完成 可以传送门ELK系列（二） Kafka集群3.4.0搭建
Kibana搭建已完成 可见传送门ELK系列（三） Kibana8.9.2搭建

主机规划

主机名        内网地址      搭建程序

ELK8-1       192.168.0.1  elasticsearch+kafka+zookeeper
ELK8-2       192.168.0.2  elasticsearch+kafka+zookeeper
ELK8-3       192.168.0.3  elasticsearch+kafka+zookeeper
logstash1    192.168.0.4  kibana+logstash

logstash1上kibana已搭建 接下来搭建logstash用来切割日志并输出到ES

系统优化设置

jvm设置
sed -i 's/-Xms4g/-Xms4g/' /usr/local/logstash-8.9.2/config/jvm.options
sed -i 's/-Xmx4g/-Xmx4g/' /usr/local/logstash-8.9.2/config/jvm.options

system纳管
[root@logstash-1 system]# cat logstash.service
[Unit]
Description=logstash service...
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/usr/local/logstash-8.9.2/bin/logstash -f /usr/local/logstash-8.9.2/config/logstash.conf
Restart=always
[Install]
WantedBy=multi-user.target
[root@logstash-1 system]# pwd
/usr/lib/systemd/system

systemctl daemon-reload
systemctl enable logstash.service

Logstash通过SSL链接ES集群

参考链接：Logstash：如何连接到带有 HTTPS 访问的集群

1.ES集群开启了SSL,Logstash通过es本身的证书生成truststore.p12去链接es集群

[root@ELK8-1 elasticsearch]# ls
bin  jdk  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.asciidoc
[root@ELK8-1 elasticsearch]# ./bin/elasticsearch-keystore list 
warning: ignoring JAVA_HOME=/usr/local/jdk-20.0.1; using bundled JDK #使用es自带的jdk
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
[root@ELK8-1 elasticsearch]# ./bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
warning: ignoring JAVA_HOME=/usr/local/jdk-20.0.1; using bundled JDK
r_FBdQ9VSxasdzcyE9G9oA  #得到ES机器http证书密码
[root@ELK8-1 certs]# keytool -keystore http.p12 -list 
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

http, Sep 11, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): FA:51:88:83:1F:64:DA:CA:C5:55:30:F9:0F:55:DF:4D:ED:0F:A2:7E:8A:4F:92:11:CE:E8:3F:75:5A:EC:DB:9B
http_ca, Sep 11, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): C6:8C:4B:34:9E:2A:AC:2E:B8:8C:CC:B0:8F:24:A2:AB:6F:52:31:A9:E0:88:80:3A:80:EB:0C:B5:70:1D:81:DB
[root@ELK8-1 certs]# keytool -import -file http_ca.crt -keystore truststore.p12 -storepass 123456@8888 -noprompt -storetype pkcs12 #将ca证书导入到truststore.p12
Certificate was added to keystore
[root@ELK8-1 certs]# keytool -keystore truststore.p12 -list 
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

mykey, Sep 12, 2023, trustedCertEntry, #mykey与es的http_ca一致 logstash可通过该信任库连接
Certificate fingerprint (SHA-256): C6:8C:4B:34:9E:2A:AC:2E:B8:8C:CC:B0:8F:24:A2:AB:6F:52:31:A9:E0:88:80:3A:80:EB:0C:B5:70:1D:81:DB

Logstash配置文件

logstash.conf作为logstash的主配置文件定义输入 filter对数据处理 output输出

[root@logstash-1 config]# pwd
/usr/local/logstash-8.9.2/config
[root@logstash-1 config]# ls
jvm.options  log4j2.properties  logstash.conf  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options  truststore.p12

[root@logstash-1 config]# vim logstash.conf
input {
  kafka {
    type => "waf" #将这个topic的日志打上type为后续输出判断
    topics => "xiamenwaf"
    decorate_events => true  #携带元数据
    bootstrap_servers => "192.168.0.1:9092, 192.168.0.2:9092, 192.168.0.3:9092"#kafka端口
  }

  kafka {
   type => "bournesu"
   topics => "bournesu.cc"
   decorate_events => true
   bootstrap_servers => "192.168.0.1:9092, 192.168.0.2:9092, 192.168.0.3:9092"
}

}

filter {
  grok {
    match => {   
      "message" => '%{IPORHOST:client.ip} %{HOSTNAME:domain} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{NOTSPACE:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code} %{NUMBER:response_size} "%{DATA:http_referer}" "%{DATA:http_user_agent}" "\[%{WORD:http.request.method} %{NOTSPACE:url.path} HTTP/%{NUMBER:http.version} %{GREEDYDATA:http.headers}\]"'
    }
  }

#grok调试可以采用kibana中的Grok Debugger 下文讲解


  date {    #日期格式转化
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] 
    target => "@timestamp"
  }

  geoip {   #geo插件对client.ip字段做地理位置判断 新增coordinates后续es映射为geo_point
    source => "client.ip"
    target => "geoip"
    add_field => ["[coordinates]", "%{[geoip][geo][location][lon]}" ] #添加经纬度
    add_field => ["[coordinates]", "%{[geoip][geo][location][lat]}" ]
  }

  mutate {  #修改字段为浮点型
  convert => [ "[coordinates]", "float" ]
  }

  mutate {  #表示来源logstash机器
    add_field => { "logstash_hostname" => "logstash-1" }
  }

  useragent {  #对UA做细分
    source => "http_user_agent"
    target => "user_agent_details"
  }

  kv {   #键值对拆分  "  "作为分隔符 ": "作为键值对分隔符 分割后字段新增前缀http.headers.
    source => "http.headers"
    field_split_pattern => "  "
    value_split => ": "
    prefix => "http.headers."

  }

  mutate {  #移除字段
            remove_field => ["http.headers"]
            remove_field => ["http.headers.Host"]
            remove_field => ["http.headers.Referer"]
            remove_field => ["http.headers.User-Agent"]
            remove_field => ["http.request.method"]
            remove_field => ["http.version"]
            remove_field => ["url.path"]
            remove_field => ["[geoip][geo][location][lon]"]
            remove_field => ["[geoip][geo][location][lat]"]
            remove_field => ["[geoip][ip]"]
            remove_field => ["timestamp"]
            remove_field => ["event"]
  }


}

output {
  if [type] == "waf" {  #判断type输出到es哪个索引
    elasticsearch {
      hosts => ["https://192.168.0.1:9200", "https://192.168.0.2:9200", "https://192.168.0.3:9200"]
      index => 'xiamenwaf_%{+yyyy-MM-dd}' #指定logstash写入的索引名
      user => "elastic"                   #指定logstash输出用户 写入权限参考下文链接
      password => "tIxjmD8nWabee6Rs0QRm"  
      ssl_certificate_verification => true
      truststore => "/usr/local/logstash-8.9.2/config/truststore.p12" #指定truststore.p12
      truststore_password => "123456@8888"
    }
  }
  if [type] == "bournesu" {
    elasticsearch {
      hosts => ["https://192.168.0.1:9200", "https://192.168.0.2:9200", "https://192.168.0.3:9200"]
      index => 'bournesu_%{+yyyy-MM-dd}'
      user => "elastic"
      password => "tIxjmD8nWabee6Rs0QRm"
      ssl_certificate_verification => true
      truststore => "/usr/local/logstash-8.9.2/config/truststore.p12"
      truststore_password => "123456@8888"
    }
  }
}

[root@logstash-1 config]# cat logstash.yml | grep -v "^#"  #logstash.yml主要定义全局参数
path.data: /data/logstash/data 
pipeline.workers: 2  #线程数与cpu核数保持一致
pipeline.batch.size: 500
pipeline.batch.delay: 5
path.logs: /data/logstash/logs #定义日志目录

Grok匹配日志切割

1.kibana开发工具Grok Debugger实时匹配测试

Logstash本身有预定义字段来匹配日志如下：

样例日志:
192.168.1.100 www.bournesu.cc - - [16/Jan/2024:15:27:55 +0800] "GET /game/sound/xsmj/special/girl/tong/8-1d6cd8abd2.mp3 HTTP/1.1" 304 0 "https://www.bournesu.cc/game/?gid=xsq12h10&fid=142401161512001&yid=pq08&yName=Gong.l.l%F0%9F%91%91&timestamp=1705389132733" "Mozilla/5.0 (Linux; Android 13; SM-F7210 Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36 XWEB/1160027 MMWEBSDK/20231201 MMWEBID/6471 MicroMessenger/8.0.45.2521(0x28002D38) WeChat/arm64 Weixin NetType/5G Language/zh_CN ABI/arm64" "[GET /game/sound/xsmj/special/girl/tong/8-1d6cd8abd2.mp3 HTTP/1.1  Host: www.bournesu.cc  Connection: keep-alive  User-Agent: Mozilla/5.0 (Linux; Android 13; SM-F7210 Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36 XWEB/1160027 MMWEBSDK/20231201 MMWEBID/6471 MicroMessenger/8.0.45.2521(0x28002D38) WeChat/arm64 Weixin NetType/5G Language/zh_CN ABI/arm64  Accept: */*  X-Requested-With: com.tencent.mm  Sec-Fetch-Site: same-origin  Sec-Fetch-Mode: cors  Sec-Fetch-Dest: empty  Referer: https://wx.fengshunhonghuo.top/game/?gid=xsq12h10&fid=142401161512001&yid=pq08&yName=Gong.l.l%F0%9F%91%91&timestamp=1705389132733  Accept-Encoding: gzip, deflate  Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7  Cookie: YCY_GAME_PRIVACY=bjPAP; KcgGameUserToken=34844ad3f2464f51bd0099f634547e84  If-None-Match: \x22658bacd8-2f61\x22  If-Modified-Since: Wed, 27 Dec 2023 04:49:28 GMT  action: ACCEPT  rule_id: 0  reason: RULE  status: 304]"

%{IPORHOST:client.ip} %{HOSTNAME:domain} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code} %{NUMBER:response_size} "%{DATA:http_referer}" "%{DATA:http_user_agent}" "\[%{WORD:http.request.method} %{URIPATHPARAM:url.path} HTTP/%{NUMBER:http.version} %{GREEDYDATA:http.headers}\]"

IPORHOST字段匹配到192.168.1.100赋值给client.ip 日志中存在空格字段grok匹配同样空格
HOSTNAME字段匹配www.bournesu.cc赋值给domain 匹配方式见logstash预定义字段
[]需要转义\ 
GREEDYDATA 代表贪婪匹配 匹配后续所有日志

2.Logstash预定义字段

logstach 预定义字段

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
 
POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
HOST %{HOSTNAME}
IPORHOST (?:%{HOSTNAME}|%{IP})
HOSTPORT %{IPORHOST}:%{POSINT}
# paths
PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox
# doesn't turn into %XX
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
# Months: January, Feb, 3, 03, 12, December
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
MONTHNUM (?:0?[1-9]|1[0-2])
MONTHNUM2 (?:0[1-9]|1[0-2])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
# Days: Monday, Tue, Thu, etc...
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
# Years?
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
# Syslog Dates: Month Day HH:MM:SS
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG (?:[\w._/%-]+)
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
# Shortcuts
QS %{QUOTEDSTRING}
# Log formats
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
# Log Levels
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

Logstash写入权限参考

logstash如无法创建索引参考logstash连接ES创建索引

Elasticsearch索引模板

Logstah切割出的字段写入索引时没有指定模板 会根据默认模板将字段映射不利于后期kibana绘图分析
启动logstash之前先在kibana指定索引所采用的索引模板

http-log字段映射配置文件如下对应上文的样例日志

{
  "_meta": {
    "version": "8.0.1"
  },
  "dynamic_templates": [
    {
      "strings_as_keyword": {
        "mapping": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "match_mapping_type": "string"
      }
    }
  ],
  "date_detection": false,
  "properties": {
    "response_code": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "BPS": {
      "type": "integer"
    },
    "auth": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "ident": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "user_agent_details": {
      "type": "object",
      "properties": {
        "os": {
          "type": "object",
          "properties": {
            "name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "version": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "full": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "name": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "device": {
          "type": "object",
          "properties": {
            "name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "type": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "response_size": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "http_user_agent": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "PPS": {
      "coerce": true,
      "index": true,
      "ignore_malformed": false,
      "store": false,
      "type": "integer",
      "doc_values": true
    },
    "http_method": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "@version": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "logstash_hostname": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "client": {
      "type": "object",
      "properties": {
        "ip": {
          "type": "ip"
        }
      }
    },
    "value": {
      "type": "double"
    },
    "timestamp": {
      "index": true,
      "ignore_malformed": false,
      "store": false,
      "type": "date",
      "doc_values": true
    },
    "geoip": {
      "type": "object",
      "properties": {
        "geo": {
          "type": "object",
          "properties": {
            "region_iso_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "city_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "country_iso_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "timezone": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "country_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "continent_code": {
              "type": "keyword"
            },
            "region_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "postal_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "ip": {
          "index": true,
          "store": false,
          "type": "ip",
          "doc_values": true
        },
        "mmdb": {
          "type": "object",
          "properties": {
            "dma_code": {
              "type": "long"
            }
          }
        }
      }
    },
    "coordinates": {
      "ignore_malformed": false,
      "type": "geo_point",
      "ignore_z_value": true
    },
    "http_version": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "message": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "url": {
      "type": "object",
      "properties": {
        "path": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "@timestamp": {
      "type": "date"
    },
    "http_referer": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "domain": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "http": {
      "type": "object",
      "properties": {
        "headers": {
          "dynamic": true,
          "type": "object",
          "enabled": true,
          "properties": {
            "Origin": {
              "type": "keyword"
            },
            "reason": {
              "type": "keyword"
            },
            "User-Agent": {
              "type": "keyword"
            },
            "Sec-Fetch-Dest": {
              "type": "keyword"
            },
            "Accept-Encoding": {
              "type": "keyword"
            },
            "DNT": {
              "type": "keyword"
            },
            "Sec-Fetch-Mode": {
              "type": "keyword"
            },
            "sec-ch-ua-mobile": {
              "type": "keyword"
            },
            "X-BD-QUIC": {
              "type": "keyword"
            },
            "Apn-Type": {
              "type": "keyword"
            },
            "Upgrade-Insecure-Requests": {
              "type": "keyword"
            },
            "action": {
              "type": "keyword"
            },
            "Sec-Fetch-User": {
              "type": "keyword"
            },
            "X-From-H3-TRNet": {
              "type": "keyword"
            },
            "Content-Length": {
              "type": "keyword"
            },
            "Content-Type": {
              "type": "keyword"
            },
            "X-TurboNet-Info": {
              "type": "keyword"
            },
            "Cookie": {
              "type": "keyword"
            },
            "If-Range": {
              "type": "keyword"
            },
            "Q-UA2": {
              "type": "keyword"
            },
            "Accept": {
              "type": "keyword"
            },
            "X-Requested-With": {
              "type": "keyword"
            },
            "Queen": {
              "type": "keyword"
            },
            "X-BDBoxApp-NetEngine": {
              "type": "keyword"
            },
            "Connection": {
              "type": "keyword"
            },
            "Referer": {
              "type": "keyword"
            },
            "Sec-Fetch-Site": {
              "type": "keyword"
            },
            "Host": {
              "type": "keyword"
            },
            "Pragma": {
              "type": "keyword"
            },
            "Range": {
              "type": "keyword"
            },
            "If-None-Match": {
              "type": "keyword"
            },
            "rule_id": {
              "type": "keyword"
            },
            "x-wap-profile": {
              "type": "keyword"
            },
            "sec-ch-ua": {
              "type": "keyword"
            },
            "Cache-Control": {
              "type": "keyword"
            },
            "X-Bd-Traceid": {
              "type": "keyword"
            },
            "sec-ch-ua-platform": {
              "type": "keyword"
            },
            "If-Modified-Since": {
              "type": "keyword"
            },
            "Accept-Language": {
              "type": "keyword"
            },
            "status": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "type": "object",
          "properties": {
            "method": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "request_path": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    }
  }
}

诉光海
关注
专栏目录
Logstash8.4在Linux系统上的安装以及配置Tomcat日志(ELK安装part2)(未完待续)
yangkei
09-07 1146
Logstash8.4在Linux系统上的安装以及配置Tomcat日志
LogStash~LogStash8的安装(包含jdk17的安装)
feizuiku0116的博客
04-18 3199
一、Java的安装 卸载虚拟机自带的Javarpm -qa | grep -i java | xargs -n1 rpm -e --nodeps 上传java安装包到/export/server/目录下 解压java压缩包tar -zxvf /export/server/jdk-8u65-linux-x64.tar.gz 删除安装包rm -rf /export/server/jdk-8u65-linux-x64.tar.gz 重命名文件夹mv /export/server/jdk1.8.0
Docker部署ELK 8.8.2（4）-logstash部署和使用
weixin_48244640的博客
07-07 1758
Docker部署ELK 8.8.2，ELK介绍，Elasticsearch、Kibana、Logstash、Filebeat安装及使用。
ELK系列（一） ElasticSearch 8.9.2集群搭建
BourneSu的博客
12-29 1205
ElasticSearch集群搭建
ELk安装部署
LT19990304的博客
08-10 195
ELK由ElasticSearch、Logstash、Kibana三个根据组成。ES是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。Logstash是一个完全开源的工具，可以对日志进行收集、分析、并将其存储供以后使用。kibana也是一个开源和免费的工具，他Kibana可以为Logstash和ES提供的日志分析友好的Web界面，可以帮助您汇总、分析和搜索重要数据日志。
Liunx 搭建 Elk 集群配置 之 Logstash 搭建
m0_46937429的博客
02-06 1167
我的版本是 logstash-7.11.0-linux-x86_64.tar.gz。在bin 目录下（当然也可以在别的目录 我是在bin目录下）输入aaa 发现安装成功。
linu 搭建ELK(es+kibana+logstash)安装包
12-02
ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat，它是一个轻量级的日志收集处理工具(Agent)，Filebeat占用资源少，适合于在各个服务器上搜集日志后...
不得不学，从零到一搭建ELK日志，Docker环境下部署 logstash工具
Java_zhujia的博客
11-14 482
Logstash是具有实时流水线能力的开源的数据收集引擎。Logstash可以动态统一不同来源的数据，并将数据标准化到您选择的目标输出。它提供了大量插件，可帮助我们解析，丰富，转换和缓冲任何类型的数据。我们主要完成在Docker环境下部署 logstash 日志收集工具，他是搭建ELK日志非常重要的一部分，上一篇文章Filebeat日志收集完成之后，将数据写入 Elasticsearch 后用 Kibana 进行可视化展示，现在我们已经完成了。
ELK日志平台（elasticsearch+logstash+kibana）搭建
用来当笔记本
12-13 2202
为了实现分布式日志数据统一收集,实现集中式查询和管理 故障排查 安全信息和事件管理本文仅仅简单介绍了ELK的安装，而ELK其他强大的功能需要继续学习。
ELK日志搜集
猪之仆的博客
12-29 573
1.ELK简介 ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat，它是一个轻量级的日志收集处理工具(Agent)，Filebeat占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具。 Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，..
ELK详解（八）——Logstash收集系统日志实战
永远是少年
04-05 7079
今天继续给大家介绍Linux运维相关知识，本文主要内容是Logstash收集系统日志。 一、Logstash配置 二、Elasticsearch结果查看 三、Kibana结果查看
elasticsearch8版本以上使用logstash同步数据
火星coder的博客
01-06 1058
elasticsearch8版本以上使用logstash同步数据
ELK 企业级日志分析系统
Jahony的博客
07-20 402
ELK 企业级日志分析系统一、ELK 概述1、ELK简介ElasticSearchLogstashKiabanaFilebeat2、为什么要使用 ELK3、完整日志系统基本特征4、ELK 的工作原理二、ELK Elasticsearch 集群部署环境准备1、在Node1、Node2节点上部署elasticsearch2、安装 Elasticsearch-head 插件ELK Logstash 部署（在 Apache 节点上操作）1．更改主机名2．安装Apahce服务（httpd）3．安装Java环境4．安装
Logstash8.3安装及试用
征途无悔
07-09 1346
logstash8.3及filebeat的搭建及测试使用
ELK本地虚拟机部署（8.12版本）
最新发布
qq_42235364的博客
04-08 906
鉴于公司服务器上日志越来越多，需要搭建ELK对日志进行采集和分析，用于后续故障排查。程序下载地址：elasticsearch:Elasticsearch 8.12.0 | Elasticlogstash:Logstash 8.12.0 | Elastickibana:Kibana 8.12.0 | Elasticfilebeat:Filebeat 8.12.0 | Elastic 1.2 环境变量（elasticsearch集群所有节点） 1.3 程序配置 1.3.1 节点1配置 elast
Logstash部署与使用
qq_19283249的博客
06-04 9090
Logstash是一个收集与处理数据的引擎，就像ElasticSearch是专门用来检索的引擎一样，Logstash用于收集、处理和转换各种数据源（文件、数据库、网站等）的数据，并将其转换为统一的格式。Logstash支持多种插件，进行数据过滤、转换和输出，可以方便地与 ES 和 Kibana 集成使用。还支持多线程处理和事件模型，可以在大规模数据处理场景下提供高性能、高可用的服务。
【Logstash8.1.0&kafka单机部署】
daiwenshuai的博客
08-03 898
Logstash8.1.0&kafka单机部署
部署logstash
Flj12138的博客
11-03 230
指定配置文件目录，rpm方式安装时，默认路径为"/usr/share/logstash/config".# 指定从文件的开头读取数据，该值仅在第一次采集是有效，且该文件在.sincedb文件中未被记录。指定数据目录，rpm方式安装默认存放路径"/usr/share/logstash/data"指定日志的输出格式，可以使json或者plain，默认值为"plain"(文本类型)。基于命令行指定Pipline ID，如果不指定，则默认值为"main"。支持文件的热加载功能。指定配置文件或配置文件的目录。
linux elk filebeat和logstash
04-30
ELK是指Elasticsearch、Logstash和Kibana，是一套常用的开源日志分析解决方案。Filebeat和Logstash是ELK中的两个组件。 Filebeat是一个开源的日志数据收集器，它可以帮助我们从各种源（如文件、系统日志等）中收集日志数据，并将其传输到指定的目的地（如Elasticsearch或Logstash）。Filebeat的作用是将日志数据收集并送到Logstash或Elasticsearch等组件中进行处理。 Logstash是一个开源的数据收集引擎，它可以帮助我们从各种数据源（如文件、消息队列等）中收集数据，并将其转换成统一的格式。Logstash的作用是接收来自Filebeat等组件传来的日志数据，并对其进行处理、转换和过滤，然后将其发送到Elasticsearch中进行存储和分析。 总的来说，Filebeat和Logstash都是ELK中用来收集和处理日志数据的组件，它们可以协同工作，提供可靠的数据收集和分析解决方案。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值