ELK系列（四） Logstash8.9.2搭建

已于 2024-01-16 16:57:29 修改
阅读量473 收藏 12
点赞数 8
分类专栏： ELK 文章标签： elk
于 2024-01-16 16:52:46 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/BourneSu/article/details/135504094
版权

ELK系列传送门

ELK系列（一） ElasticSearch 8.9.2集群搭建

ELK系列（二）Kafka集群3.4.0搭建

ELK系列（三） Kibana8.9.2搭建

ELK系列（四） Logstash8.9.2搭建

ELK数据流规划

ELK接入nginx-acces日志数据流规划


数据流向
		filebeat -------> kafka ----> logstash ----> ES --------> kibana
		 收集              mq          处理日志     存储、搜索     展示、分析
                                                       |
										               |
								                   elastalert2---钉钉告警


ES集群搭建已完成 可见传送门ELK系列（一） ElasticSearch 8.9.2集群搭建
Kafka集群搭建已完成 可以传送门ELK系列（二） Kafka集群3.4.0搭建
Kibana搭建已完成 可见传送门ELK系列（三） Kibana8.9.2搭建

主机规划

主机名        内网地址      搭建程序

ELK8-1       192.168.0.1  elasticsearch+kafka+zookeeper
ELK8-2       192.168.0.2  elasticsearch+kafka+zookeeper
ELK8-3       192.168.0.3  elasticsearch+kafka+zookeeper
logstash1    192.168.0.4  kibana+logstash

logstash1上kibana已搭建 接下来搭建logstash用来切割日志并输出到ES

系统优化设置

jvm设置
sed -i 's/-Xms4g/-Xms4g/' /usr/local/logstash-8.9.2/config/jvm.options
sed -i 's/-Xmx4g/-Xmx4g/' /usr/local/logstash-8.9.2/config/jvm.options

system纳管
[root@logstash-1 system]# cat logstash.service
[Unit]
Description=logstash service...
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/usr/local/logstash-8.9.2/bin/logstash -f /usr/local/logstash-8.9.2/config/logstash.conf
Restart=always
[Install]
WantedBy=multi-user.target
[root@logstash-1 system]# pwd
/usr/lib/systemd/system

systemctl daemon-reload
systemctl enable logstash.service

Logstash通过SSL链接ES集群

参考链接：Logstash：如何连接到带有 HTTPS 访问的集群

1.ES集群开启了SSL,Logstash通过es本身的证书生成truststore.p12去链接es集群

[root@ELK8-1 elasticsearch]# ls
bin  jdk  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.asciidoc
[root@ELK8-1 elasticsearch]# ./bin/elasticsearch-keystore list 
warning: ignoring JAVA_HOME=/usr/local/jdk-20.0.1; using bundled JDK #使用es自带的jdk
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
[root@ELK8-1 elasticsearch]# ./bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
warning: ignoring JAVA_HOME=/usr/local/jdk-20.0.1; using bundled JDK
r_FBdQ9VSxasdzcyE9G9oA  #得到ES机器http证书密码
[root@ELK8-1 certs]# keytool -keystore http.p12 -list 
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

http, Sep 11, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): FA:51:88:83:1F:64:DA:CA:C5:55:30:F9:0F:55:DF:4D:ED:0F:A2:7E:8A:4F:92:11:CE:E8:3F:75:5A:EC:DB:9B
http_ca, Sep 11, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): C6:8C:4B:34:9E:2A:AC:2E:B8:8C:CC:B0:8F:24:A2:AB:6F:52:31:A9:E0:88:80:3A:80:EB:0C:B5:70:1D:81:DB
[root@ELK8-1 certs]# keytool -import -file http_ca.crt -keystore truststore.p12 -storepass 123456@8888 -noprompt -storetype pkcs12 #将ca证书导入到truststore.p12
Certificate was added to keystore
[root@ELK8-1 certs]# keytool -keystore truststore.p12 -list 
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

mykey, Sep 12, 2023, trustedCertEntry, #mykey与es的http_ca一致 logstash可通过该信任库连接
Certificate fingerprint (SHA-256): C6:8C:4B:34:9E:2A:AC:2E:B8:8C:CC:B0:8F:24:A2:AB:6F:52:31:A9:E0:88:80:3A:80:EB:0C:B5:70:1D:81:DB

Logstash配置文件

logstash.conf作为logstash的主配置文件定义输入 filter对数据处理 output输出

[root@logstash-1 config]# pwd
/usr/local/logstash-8.9.2/config
[root@logstash-1 config]# ls
jvm.options  log4j2.properties  logstash.conf  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options  truststore.p12

[root@logstash-1 config]# vim logstash.conf
input {
  kafka {
    type => "waf" #将这个topic的日志打上type为后续输出判断
    topics => "xiamenwaf"
    decorate_events => true  #携带元数据
    bootstrap_servers => "192.168.0.1:9092, 192.168.0.2:9092, 192.168.0.3:9092"#kafka端口
  }

  kafka {
   type => "bournesu"
   topics => "bournesu.cc"
   decorate_events => true
   bootstrap_servers => "192.168.0.1:9092, 192.168.0.2:9092, 192.168.0.3:9092"
}

}

filter {
  grok {
    match => {   
      "message" => '%{IPORHOST:client.ip} %{HOSTNAME:domain} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{NOTSPACE:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code} %{NUMBER:response_size} "%{DATA:http_referer}" "%{DATA:http_user_agent}" "\[%{WORD:http.request.method} %{NOTSPACE:url.path} HTTP/%{NUMBER:http.version} %{GREEDYDATA:http.headers}\]"'
    }
  }

#grok调试可以采用kibana中的Grok Debugger 下文讲解


  date {    #日期格式转化
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] 
    target => "@timestamp"
  }

  geoip {   #geo插件对client.ip字段做地理位置判断 新增coordinates后续es映射为geo_point
    source => "client.ip"
    target => "geoip"
    add_field => ["[coordinates]", "%{[geoip][geo][location][lon]}" ] #添加经纬度
    add_field => ["[coordinates]", "%{[geoip][geo][location][lat]}" ]
  }

  mutate {  #修改字段为浮点型
  convert => [ "[coordinates]", "float" ]
  }

  mutate {  #表示来源logstash机器
    add_field => { "logstash_hostname" => "logstash-1" }
  }

  useragent {  #对UA做细分
    source => "http_user_agent"
    target => "user_agent_details"
  }

  kv {   #键值对拆分  "  "作为分隔符 ": "作为键值对分隔符 分割后字段新增前缀http.headers.
    source => "http.headers"
    field_split_pattern => "  "
    value_split => ": "
    prefix => "http.headers."

  }

  mutate {  #移除字段
            remove_field => ["http.headers"]
            remove_field => ["http.headers.Host"]
            remove_field => ["http.headers.Referer"]
            remove_field => ["http.headers.User-Agent"]
            remove_field => ["http.request.method"]
            remove_field => ["http.version"]
            remove_field => ["url.path"]
            remove_field => ["[geoip][geo][location][lon]"]
            remove_field => ["[geoip][geo][location][lat]"]
            remove_field => ["[geoip][ip]"]
            remove_field => ["timestamp"]
            remove_field => ["event"]
  }


}

output {
  if [type] == "waf" {  #判断type输出到es哪个索引
    elasticsearch {
      hosts => ["https://192.168.0.1:9200", "https://192.168.0.2:9200", "https://192.168.0.3:9200"]
      index => 'xiamenwaf_%{+yyyy-MM-dd}' #指定logstash写入的索引名
      user => "elastic"                   #指定logstash输出用户 写入权限参考下文链接
      password => "tIxjmD8nWabee6Rs0QRm"  
      ssl_certificate_verification => true
      truststore => "/usr/local/logstash-8.9.2/config/truststore.p12" #指定truststore.p12
      truststore_password => "123456@8888"
    }
  }
  if [type] == "bournesu" {
    elasticsearch {
      hosts => ["https://192.168.0.1:9200", "https://192.168.0.2:9200", "https://192.168.0.3:9200"]
      index => 'bournesu_%{+yyyy-MM-dd}'
      user => "elastic"
      password => "tIxjmD8nWabee6Rs0QRm"
      ssl_certificate_verification => true
      truststore => "/usr/local/logstash-8.9.2/config/truststore.p12"
      truststore_password => "123456@8888"
    }
  }
}

[root@logstash-1 config]# cat logstash.yml | grep -v "^#"  #logstash.yml主要定义全局参数
path.data: /data/logstash/data 
pipeline.workers: 2  #线程数与cpu核数保持一致
pipeline.batch.size: 500
pipeline.batch.delay: 5
path.logs: /data/logstash/logs #定义日志目录

Grok匹配日志切割

1.kibana开发工具Grok Debugger实时匹配测试

Logstash本身有预定义字段来匹配日志如下：

样例日志:
192.168.1.100 www.bournesu.cc - - [16/Jan/2024:15:27:55 +0800] "GET /game/sound/xsmj/special/girl/tong/8-1d6cd8abd2.mp3 HTTP/1.1" 304 0 "https://www.bournesu.cc/game/?gid=xsq12h10&fid=142401161512001&yid=pq08&yName=Gong.l.l%F0%9F%91%91&timestamp=1705389132733" "Mozilla/5.0 (Linux; Android 13; SM-F7210 Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36 XWEB/1160027 MMWEBSDK/20231201 MMWEBID/6471 MicroMessenger/8.0.45.2521(0x28002D38) WeChat/arm64 Weixin NetType/5G Language/zh_CN ABI/arm64" "[GET /game/sound/xsmj/special/girl/tong/8-1d6cd8abd2.mp3 HTTP/1.1  Host: www.bournesu.cc  Connection: keep-alive  User-Agent: Mozilla/5.0 (Linux; Android 13; SM-F7210 Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36 XWEB/1160027 MMWEBSDK/20231201 MMWEBID/6471 MicroMessenger/8.0.45.2521(0x28002D38) WeChat/arm64 Weixin NetType/5G Language/zh_CN ABI/arm64  Accept: */*  X-Requested-With: com.tencent.mm  Sec-Fetch-Site: same-origin  Sec-Fetch-Mode: cors  Sec-Fetch-Dest: empty  Referer: https://wx.fengshunhonghuo.top/game/?gid=xsq12h10&fid=142401161512001&yid=pq08&yName=Gong.l.l%F0%9F%91%91&timestamp=1705389132733  Accept-Encoding: gzip, deflate  Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7  Cookie: YCY_GAME_PRIVACY=bjPAP; KcgGameUserToken=34844ad3f2464f51bd0099f634547e84  If-None-Match: \x22658bacd8-2f61\x22  If-Modified-Since: Wed, 27 Dec 2023 04:49:28 GMT  action: ACCEPT  rule_id: 0  reason: RULE  status: 304]"

%{IPORHOST:client.ip} %{HOSTNAME:domain} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{URIPATHPARAM:request_path} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code} %{NUMBER:response_size} "%{DATA:http_referer}" "%{DATA:http_user_agent}" "\[%{WORD:http.request.method} %{URIPATHPARAM:url.path} HTTP/%{NUMBER:http.version} %{GREEDYDATA:http.headers}\]"

IPORHOST字段匹配到192.168.1.100赋值给client.ip 日志中存在空格字段grok匹配同样空格
HOSTNAME字段匹配www.bournesu.cc赋值给domain 匹配方式见logstash预定义字段
[]需要转义\ 
GREEDYDATA 代表贪婪匹配 匹配后续所有日志

2.Logstash预定义字段

logstach 预定义字段

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
 
POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
HOST %{HOSTNAME}
IPORHOST (?:%{HOSTNAME}|%{IP})
HOSTPORT %{IPORHOST}:%{POSINT}
# paths
PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox
# doesn't turn into %XX
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
# Months: January, Feb, 3, 03, 12, December
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
MONTHNUM (?:0?[1-9]|1[0-2])
MONTHNUM2 (?:0[1-9]|1[0-2])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
# Days: Monday, Tue, Thu, etc...
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
# Years?
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
# Syslog Dates: Month Day HH:MM:SS
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG (?:[\w._/%-]+)
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
# Shortcuts
QS %{QUOTEDSTRING}
# Log formats
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
# Log Levels
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

Logstash写入权限参考

logstash如无法创建索引参考logstash连接ES创建索引

Elasticsearch索引模板

Logstah切割出的字段写入索引时没有指定模板 会根据默认模板将字段映射不利于后期kibana绘图分析
启动logstash之前先在kibana指定索引所采用的索引模板

http-log字段映射配置文件如下对应上文的样例日志

{
  "_meta": {
    "version": "8.0.1"
  },
  "dynamic_templates": [
    {
      "strings_as_keyword": {
        "mapping": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "match_mapping_type": "string"
      }
    }
  ],
  "date_detection": false,
  "properties": {
    "response_code": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "BPS": {
      "type": "integer"
    },
    "auth": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "ident": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "user_agent_details": {
      "type": "object",
      "properties": {
        "os": {
          "type": "object",
          "properties": {
            "name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "version": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "full": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "name": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "device": {
          "type": "object",
          "properties": {
            "name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "type": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "response_size": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "http_user_agent": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "PPS": {
      "coerce": true,
      "index": true,
      "ignore_malformed": false,
      "store": false,
      "type": "integer",
      "doc_values": true
    },
    "http_method": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "@version": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "logstash_hostname": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "client": {
      "type": "object",
      "properties": {
        "ip": {
          "type": "ip"
        }
      }
    },
    "value": {
      "type": "double"
    },
    "timestamp": {
      "index": true,
      "ignore_malformed": false,
      "store": false,
      "type": "date",
      "doc_values": true
    },
    "geoip": {
      "type": "object",
      "properties": {
        "geo": {
          "type": "object",
          "properties": {
            "region_iso_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "city_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "country_iso_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "timezone": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "country_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "continent_code": {
              "type": "keyword"
            },
            "region_name": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            },
            "postal_code": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "ip": {
          "index": true,
          "store": false,
          "type": "ip",
          "doc_values": true
        },
        "mmdb": {
          "type": "object",
          "properties": {
            "dma_code": {
              "type": "long"
            }
          }
        }
      }
    },
    "coordinates": {
      "ignore_malformed": false,
      "type": "geo_point",
      "ignore_z_value": true
    },
    "http_version": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "message": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "url": {
      "type": "object",
      "properties": {
        "path": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "@timestamp": {
      "type": "date"
    },
    "http_referer": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "domain": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    },
    "http": {
      "type": "object",
      "properties": {
        "headers": {
          "dynamic": true,
          "type": "object",
          "enabled": true,
          "properties": {
            "Origin": {
              "type": "keyword"
            },
            "reason": {
              "type": "keyword"
            },
            "User-Agent": {
              "type": "keyword"
            },
            "Sec-Fetch-Dest": {
              "type": "keyword"
            },
            "Accept-Encoding": {
              "type": "keyword"
            },
            "DNT": {
              "type": "keyword"
            },
            "Sec-Fetch-Mode": {
              "type": "keyword"
            },
            "sec-ch-ua-mobile": {
              "type": "keyword"
            },
            "X-BD-QUIC": {
              "type": "keyword"
            },
            "Apn-Type": {
              "type": "keyword"
            },
            "Upgrade-Insecure-Requests": {
              "type": "keyword"
            },
            "action": {
              "type": "keyword"
            },
            "Sec-Fetch-User": {
              "type": "keyword"
            },
            "X-From-H3-TRNet": {
              "type": "keyword"
            },
            "Content-Length": {
              "type": "keyword"
            },
            "Content-Type": {
              "type": "keyword"
            },
            "X-TurboNet-Info": {
              "type": "keyword"
            },
            "Cookie": {
              "type": "keyword"
            },
            "If-Range": {
              "type": "keyword"
            },
            "Q-UA2": {
              "type": "keyword"
            },
            "Accept": {
              "type": "keyword"
            },
            "X-Requested-With": {
              "type": "keyword"
            },
            "Queen": {
              "type": "keyword"
            },
            "X-BDBoxApp-NetEngine": {
              "type": "keyword"
            },
            "Connection": {
              "type": "keyword"
            },
            "Referer": {
              "type": "keyword"
            },
            "Sec-Fetch-Site": {
              "type": "keyword"
            },
            "Host": {
              "type": "keyword"
            },
            "Pragma": {
              "type": "keyword"
            },
            "Range": {
              "type": "keyword"
            },
            "If-None-Match": {
              "type": "keyword"
            },
            "rule_id": {
              "type": "keyword"
            },
            "x-wap-profile": {
              "type": "keyword"
            },
            "sec-ch-ua": {
              "type": "keyword"
            },
            "Cache-Control": {
              "type": "keyword"
            },
            "X-Bd-Traceid": {
              "type": "keyword"
            },
            "sec-ch-ua-platform": {
              "type": "keyword"
            },
            "If-Modified-Since": {
              "type": "keyword"
            },
            "Accept-Language": {
              "type": "keyword"
            },
            "status": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "type": "object",
          "properties": {
            "method": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        }
      }
    },
    "request_path": {
      "eager_global_ordinals": false,
      "norms": false,
      "index": true,
      "store": false,
      "type": "keyword",
      "index_options": "docs",
      "split_queries_on_whitespace": false,
      "doc_values": true
    }
  }
}

诉光海
关注
  • 8
    点赞
  • 踩
  • 12
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
ELK 企业级日志分析系统
Jahony的博客
07-20 357
ELK 企业级日志分析系统一、ELK 概述1、ELK简介ElasticSearchLogstashKiabanaFilebeat2、为什么要使用 ELK3、完整日志系统基本特征4、ELK 的工作原理二、ELK Elasticsearch 集群部署环境准备1、在Node1、Node2节点上部署elasticsearch2、安装 Elasticsearch-head 插件ELK Logstash 部署（在 Apache 节点上操作）1．更改主机名2．安装Apahce服务（httpd）3．安装Java环境4．安装
ELK(8)：ELK-logstash收集日志写入数据库
weixin_30758821的博客
07-16 774
ELK(8)：ELK-logstash收集日志写入数据库 在使用ELK对日志进行收集的时候，如果需要对数据进行存档，可以考虑使用数据库的方式 其实我不建议，日志真的太多了，数据库扛不住的 安装logstash的数据库插件 安装logstash的数据库插件需要先安装gem源: sudo yum install gem -y sudo gem –v #替换gem源...
ELk安装部署
LT19990304的博客
08-10 140
ELK由ElasticSearch、Logstash、Kibana三个根据组成。ES是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。Logstash是一个完全开源的工具，可以对日志进行收集、分析、并将其存储供以后使用。kibana也是一个开源和免费的工具，他Kibana可以为Logstash和ES提供的日志分析友好的Web界面，可以帮助您汇总、分析和搜索重要数据日志。
k8s部署elk8 直接通过logstash获取日志文件方式
本作者：想花
11-17 539
【代码】k8s部署elk8 直接通过logstash获取日志文件方式。
ELK详解（八）——Logstash收集系统日志实战
永远是少年
04-05 6855
今天继续给大家介绍Linux运维相关知识，本文主要内容是Logstash收集系统日志。 一、Logstash配置 二、Elasticsearch结果查看 三、Kibana结果查看
elasticsearch8版本以上使用logstash同步数据
火星coder的博客
01-06 909
elasticsearch8版本以上使用logstash同步数据
快速搭建ELK日志分析系统
01-07
官网地址:...ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。 Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等结
linu 搭建ELK(es+kibana+logstash)安装包
12-02
ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat，它是一个轻量级的日志收集处理工具(Agent)，Filebeat占用资源少，适合于在各个服务器上搜集日志后...
logstash ELK
11-05
Logstash 是一个开源的数据收集引擎，它具有备实时数据传输能力。它可以统一过滤来自不同源的数据，并按照开发者的制定的规范输出到目的地。
ELK日志分析平台搭建全过程
02-24
当生产环境有很多服务器、很多业务模块的日志需要每时每刻查看时系统：centos6.5JDK：1.8Elasticsearch-5.0.0Logstash-5.0.0kibana-5.0.01、安装JDK下载JDK：...将安装包拷贝至安装服务器/usr/local目录[root@localhost...
ELK日志中心搭建（三）- logstash8.5.3
03-04
logstash8.5.3-日志配置
8版本ELK集群部署
xhredeem的博客
12-30 1294
ELK集群搭建
logstash配置文件详解
weixin_30378311的博客
06-16 626
logstash pipeline 包含两个必须的元素：input和output，和一个可选元素：filter。 从input读取事件源，（经过filter解析和处理之后），从output输出到目标存储库（elasticsearch或其他）。 在生产环境使用logstash，一般使用都将配置写入文件里面，然后启动logstash。 具体参照官网：https://www...
Logstash8.4在Linux系统上的安装以及配置Tomcat日志(ELK安装part2)(未完待续)
yangkei
09-07 1083
Logstash8.4在Linux系统上的安装以及配置Tomcat日志
elasticsearch8.9.1部署及配置基本安全性以及安全的 HTTPS 流量
qq_43700739的博客
09-13 1877
2、在任何单个节点上，从安装 Elasticsearch 的目录中，运行 Elasticsearch 用于生成证书签名请求 （CSR） 的 HTTP 证书工具。2、修改elasticsearch配置文件elasticsearch.yml(支持ipv4以及ipv6双栈)，文件权限修改成非root的应用程序账号。注：1、输出文件名称（默认）2、CA密码设置（可留空直接回车，生产环境建议设置）输入http.p12密码(配置了的节点都需要将密码存入安全设置中)注：1、输入CA密码。注：1、2：输入密码。
docker-compose部署elk（8.9.0）并开启ssl认证
qq_41139119的博客
10-15 2092
docker-compose部署elk（8.9.0）并开启ssl认证
Docker搭建ELKF日志分析系统
weixin_73059729的博客
06-07 1427
在kibana中，默认通过时间来排序。在Management中找到Indexpatterns，单击进去可以看到类似以下图片中的界面，填写www-bdqn-cn-pro-access-*从上面的输入配置文件中可以看出，这里来采用的第一种图形，在Filebeat读取数据后，向logstash发送数据前添加www-bdqn-cn-pro-access的tag。在filter段内的第一行是判断语句，如果www-bdqn-cn-pro-access自定义字符在tags内，则使用grok段内的语句对日志进行处理。
【案例】某企业人力资源盘点知识.docx
最新发布
06-10
【案例】某企业人力资源盘点知识.docx
AUTOSAR-SWS-SAEJ1939NetworkManagement.pdf
06-10
AUTOSAR_SWS_SAEJ1939NetworkManagement.pdf
linux elk filebeat和logstash
04-30
ELK是指Elasticsearch、Logstash和Kibana，是一套常用的开源日志分析解决方案。Filebeat和Logstash是ELK中的两个组件。 Filebeat是一个开源的日志数据收集器，它可以帮助我们从各种源（如文件、系统日志等）中收集...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值