【ELKF】一款让你舍不得放手的分布式日志系统ELKF就是那么容易上手

【不好意思，本文是导入本地电脑笔记，图片无法显示，后续加上。谢谢查看。】

一、安装elasticsearch 7.9.3（9200）

1、下载

https://www.elastic.co/cn/downloads/elasticsearch

2、解压

3、配置

1、数据、日志存储路径配置

默认情况下存储到安装目录下到 data和logs目录下到。

config/elasticsearch.yml可配置：

path.data: /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/cyzhong/data

path.logs: /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/cyzhong/logs

目前保留默认目录。

4、启动

4.1、控制台运行

$ cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/
$ ./bin/elasticsearch

这种情况是在控制台运行的，一旦控制台关闭，服务就停止了。

4.2、后台运行

并输出日志：

$ nohup ./bin/elasticsearch > elasticsearch.log 2>&1 &

$ nohup /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/bin/elasticsearch > /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/logs/elasticsearch.log 2>&1 &

4.3、运行执行文件（推荐）

$ sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/start.sh

5、查看日志（推荐）

tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/logs/elasticsearch.log

6、访问测试

6.1、命令行测试访问

$ curl 127.0.0.1:9200

6.2、浏览器访问

http://127.0.0.1:9200

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JfRin6tF-1635087052692)(images/ksohtml/wps7knXIY.jpg)]

7、查看进程

$ ps -ef|grep elasticsearch

8、磁盘空间

配置：

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 6gb
cluster.routing.allocation.disk.watermark.high: 4gb

异常

[2021-10-05T14:10:34,741][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [chenyuanzhongdeMacBook-Pro.local] uncaught exception in thread [main]org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unable to consistently parse [cluster.routing.allocation.disk.watermark.low=6gb], [cluster.routing.allocation.disk.watermark.high=4gb], and [cluster.routing.allocation.disk.watermark.flood_stage=95%] as percentage or bytes

改：

#cluster.routing.allocation.disk.threshold_enabled: true
#cluster.routing.allocation.disk.watermark.low: 60gb
#cluster.routing.allocation.disk.watermark.high: 40gb
cluster.routing.allocation.disk.threshold_enabled: false

异常：

[2021-10-06T15:33:49,176][WARN ][r.suppressed ] [chenyuanzhongdeMacBook-Pro.local] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=1000, index=.kibana, from=0}org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

清除很多文件之后重启：

[2021-10-06T16:00:10,727][WARN ][r.suppressed ] [chenyuanzhongdeMacBook-Pro.local] path: /.kibana/_doc/space%3Adefault, params: {index=.kibana, id=space:default}org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][_doc][space:default]: routing [null]]

删除data数据后：

[2021-10-06T16:08:55,234][WARN ][r.suppressed ] [chenyuanzhongdeMacBook-Pro.local] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=20, index=.kibana, from=0}org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

设置：

node.name: node-1

异常：

[2021-10-06T15:33:49,176][WARN ][r.suppressed ] [chenyuanzhongdeMacBook-Pro.local] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=1000, index=.kibana, from=0}org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

重新清除data数据（前面是清除数据位置错误）：

正确路径：

/Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/cyzhong/data

正常运行：

[2021-10-06T16:20:11,311][INFO ][o.e.l.LicenseService ] [node-1] license [d635e3f4-f534-4530-b20f-10df2a5f6df4] mode [basic] - valid[2021-10-06T16:20:11,312][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled[2021-10-06T16:20:28,069][INFO ][o.e.c.m.MetadataCreateIndexService] [node-1] [.kibana] creating index, cause [auto(bulk api)], templates [], shards [1]/[1][2021-10-06T16:20:28,502][INFO ][o.e.c.m.MetadataMappingService] [node-1] [.kibana/PB3McRByS9m5v7s1V6v2Jw] create_mapping [_doc]

9、查索引

curl  'localhost:9200/_cat/indices?v'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OyHajVWQ-1635087052693)(images/ksohtml/wps4pcCKu.jpg)]

10、查health

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-B3VEzjRn-1635087052693)(images/ksohtml/wpsa1rBfP.jpg)]

curl 'http://localhost:9200/_cat/health?v'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vXrFNSRi-1635087052694)(images/ksohtml/wpsG9ZUgk.jpg)]

10.1改健康状态为100%

连接：

http://localhost:9200/_settings

数据：

{ “number_of_replicas” : 0 }

方式：put、get

put方式：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rKlDBIty-1635087052694)(images/ksohtml/wpsBTzVdy.jpg)]

get方式：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-B7BSUrsE-1635087052695)(images/ksohtml/wpsS7UHam.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-I6ieoSSG-1635087052696)(images/ksohtml/wpsf0jMhv.jpg)]

11、删除索引

启动时报异常（该索引非常重要，谨慎误删）：No shard available for

curl -XDELETE http://localhost:9200/.kibana

日志：{“acknowledged”:true}%

12、插件：本地监控工具ES-head

https://www.cnblogs.com/wangzhuxing/p/9352258.html#_label0

12.1 安装grunt

npm install -g grunt

chyzhong@chenyuanzhongdeMacBook-Pro 15-ELKF7.9.3-mac % npm install -g grunt added 101 packages in 7s 1 package is looking for funding run npm fund for detailsnpm noticenpm notice New minor version of npm available! 7.19.1 -> 7.24.2npm notice Changelog: https://github.com/npm/cli/releases/tag/v7.24.2npm notice Run npm install -g npm@7.24.2 to update!npm notice

12.2 安装grunt-cli

npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy

12.3 下载head插件源码

https://github.com/mobz/elasticsearch-head

12.4 安装head

解压后，进入elasticsearch-head-master目录：

cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-head-master
npm install

12.5 关闭ElasticSearch

elasticsearch-head-master % jps
1932 Elasticsearch
813
3119 Jps 
elasticsearch-head-master % kill 1932 

12.6 配置 ElasticSearch

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"

12.7 配置head

/Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-head-master/Gruntfile.js

connect: {
			server: {
				options: {
					port: 9100,
					base: '.',
					keepalive: true
				}
			}
		}

改为增加hostname：

connect: {
			server: {
				options: {
					hostname: '0.0.0.0',
					port: 9100,
					base: '.',
					keepalive: true
				}
			}
		}

12.8 启动ElasticSearch

$ sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/start.sh

12.9 启动head（推荐）

cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-head-master

这个是命令行启动

elasticsearch-head-master % grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

或者：

npm run start       #启动es-head
npm run start &      #后台启动es-head

或者：

grunt server       #启动es-head
nohup grunt server &   #后台启动es-head 

12.10 访问head

http://localhost:9100

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-mpo4VM1c-1635087052696)(images/ksohtml/wpsYjHwNp.jpg)]

健康值修改后，日志读取后：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WVLtFOiy-1635087052697)(images/ksohtml/wpsNm9OU9.jpg)]

13、插件：安装KOPF插件

访问：http://localhost:9200/_plugin/kopf

源码：

https://github.com/lmenezes/elasticsearch-kopf

https://blog.51cto.com/liqingbiao/1926329

https://www.cnblogs.com/xiaodai12138/p/10084465.html

安装失败：

安装失败：
git clone git://github.com/lmenezes/elasticsearch-kopf.git
cd elasticsearch-kopf
git checkout {branch|version}
npm install          ###安装失败
grunt server         ## 控制台启动

二、安装kibana7.9.3（5601）

1、下载、解压

​ [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SkJ6e0Pa-1635087052697)(images/ksohtml/wpsDfenRB.jpg)]

2、配置

kibana.yml文件内容

server.port: 5601
elasticsearch.hosts: ["http://localhost:9200"]
kibana.index: ".kibana"
# 设置中文
i18n.locale: "zh-CN"

3、启动

cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3
$ bin/kibana

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-o9UtojxT-1635087052698)(images/ksohtml/wpsTygL7s.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-O8tq12KJ-1635087052699)(images/ksohtml/wpsLvWHMb.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-bVnHxOMw-1635087052700)(images/ksohtml/wps0WvLxX.jpg)]

启动kibana前提是先要启动es，不然就更上面警告一样。还要有node环境。

3.1、命令行启动

nohup /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/bin/kibana > /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/logs/kibana.log 2>&1 &

3.2、执行文件（推荐）

将上面的命令存放到start.sh文件内。

$ sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/start.sh

4、访问

http://localhost:5601

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BEYnKqew-1635087052700)(images/ksohtml/wpsThv7Yg.jpg)]

5、查看进程

$ lsof -i:5601

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-DFbZ8Wcd-1635087052701)(images/ksohtml/wpsFE2SIF.jpg)]

6、修改端口等属性

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vuG32o0S-1635087052702)(images/ksohtml/wps9PyMcy.jpg)]

打开kibana.yml文件，修改属性：

#server.port: 5601
server.port: 5602

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-hWKmpNo7-1635087052703)(images/ksohtml/wpsKZ6QwF.jpg)]

#elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.hosts: ["http://localhost:9201"]

7、查看日志（推荐）

tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/logs/kibana.log

三、logstash-7.9.3-zip（5044）

1、下载

2、解压

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nUxGyTQ9-1635087052703)(images/ksohtml/wps9zsf6m.jpg)]

3、配置

$ cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip

3.1、配置logstash.conf（推荐）

将logstash-sample.conf拷贝一份并命名为logstash.conf。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-H4DcJqGt-1635087052704)(images/ksohtml/wpsexbHcd.jpg)]

logstash.conf这个文件目前没有用到，但其与构造一样。

后来配置：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BDXRzHvC-1635087052704)(images/ksohtml/wps7Vt4ku.jpg)]

3.2、配置first-pipeline.conf

接收filebeat传送过来的日志，输出到控制台上。

创建first-pipeline.conf，配置以下内容：

input {
  beats {
    port => 5044
  }
}
output {
   stdout { codec => rubydebug }
}

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-W8CQiIMt-1635087052705)(images/ksohtml/wpsPp4Oiz.jpg)]

3.3、检查配置first-pipeline.conf

【运行出错】

$ bin/logstash -f config/first-pipeline.conf --config.test_and_exit

Sending Logstash logs to /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logs which is now configured via log4j2.properties[2020-10-28T10:33:31,177][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“7.9.3”, “jruby.version”=>“jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot™ 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [darwin-x86_64]”}[2020-10-28T10:33:31,403][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified[2020-10-28T10:33:32,384][INFO ][org.reflections.Reflections] Reflections took 31 ms to scan 1 urls, producing 22 keys and 45 valuesConfiguration OK[2020-10-28T10:33:33,363][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

（画外音：–config.test_and_exit选项的意思是解析配置文件并报告任何错误）

【运行正常】

$ bin/logstash -f config/first-pipeline.conf --config.reload.automatic

第一次有错误（原因是logstash已经启动）：
[2020-10-28T10:37:45,201][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“7.9.3”, “jruby.version”=>“jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot™ 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [darwin-x86_64]”}[2020-10-28T10:37:45,414][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified[2020-10-28T10:37:45,428][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the “path.data” setting.[2020-10-28T10:37:45,434][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
第二次正常（先把logstash杀死进程）
Sending Logstash logs to /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logs which is now configured via log4j2.properties[2020-10-28T10:41:21,693][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“7.9.3”, “jruby.version”=>“jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot™ 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [darwin-x86_64]”}[2020-10-28T10:41:21,926][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified[2020-10-28T10:41:23,492][INFO ][org.reflections.Reflections] Reflections took 31 ms to scan 1 urls, producing 22 keys and 45 values[2020-10-28T10:41:24,586][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>“main”, “pipeline.workers”=>8, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>50, “pipeline.max_inflight”=>1000, “pipeline.sources”=>["/Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/first-pipeline.conf"], :thread=>"#<Thread:0x5692cdc run>"}[2020-10-28T10:41:25,213][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {“seconds”=>0.61}[2020-10-28T10:41:25,237][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>“0.0.0.0:5044”}[2020-10-28T10:41:25,259][INFO ][logstash.javapipeline ][main] Pipeline started {“pipeline.id”=>“main”}[2020-10-28T10:41:25,343][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2020-10-28T10:41:25,400][INFO ][org.logstash.beats.Server][main][9646b5bee08c119176779308465cf16c68567c4b6a7a1617d93f8167b365c7f5] Starting server on port: 5044[2020-10-28T10:41:25,686][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

（画外音：–config.reload.automatic选项的意思是启用自动配置加载，以至于每次你修改完配置文件以后无需停止然后重启Logstash）

4、启动

$ cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip

4.1、启动first-pipeline.conf配置

$ nohup bin/logstash -f config/first-pipeline.conf --config.reload.automatic > logs/logstash-first-pipeline.log 2>&1 &

4.2、启动logstash.conf

$ nohup ./bin/logstash > logs/logstash.log 2>&1 &

$nohup bin/logstash --config.reload.automatic -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/config/logstash.conf > logs/logstash.log 2>&1 &

4.3、执行文件（推荐）

$ sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/start.sh

5、查看日志

5.1、查看first-pipeline.conf启动的日志

tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logs/logstash-first-pipeline.log 

5.2、查看logstash.conf启动日志（推荐）

tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logs/logstash.log

5、测试

5.1、测试管道（独立测试，控制台显示）

【允许Logstash最基本的管道】先kill掉logstash进程，重新执行启动。

chyzhong@chenyuanzhongdeMacBook-Pro logstash-7.9.3-zip % bin/logstash -e 'input { stdin {} } output { stdout {} }'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Hn9uRqAM-1635087052706)(images/ksohtml/wpsRYNDu3.jpg)]

然后输入：hello world

控制台输出：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-t58irAnR-1635087052706)(images/ksohtml/wpsFoRgVp.jpg)]

6、查看进程

ps -ef|grep logstash

7、采集json格式日志

filter {
  if [filetype] == "SpringBootLogback01_info" {
    json {
      source => "message"
      #remove_field => ["message","@version","path","input","log","agent","ecs","tags"] #不需要记录，移除的字段
    }
  }
  if [filetype] == "springboot_logback01_error" {
    json {
      source => "message"
      #remove_field => ["message","@version","path","input","log","agent","ecs","tags"] #不需要记录，移除的字段
    }
  }
  if [filetype] == "nginx-json-8080" {
    json {
      source => "message"
      #remove_field => ["message","@version","path","input","log","agent","ecs","tags"] #不需要记录，移除的字段
  	}
	}
}

四、filebeat7.9.3

1、下载

2、解压

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-sWZqn6Q7-1635087052707)(images/ksohtml/wpsYaorqX.jpg)]

3、配置向es输入日志

4、配置向logstash输入日志

4.1、准备示例日志

/Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logstash-tutorial.log

4.2、配置filebeat.yml（输出日志到logstash）

filebeat.inputs:
- type: log
  enabled: true
  paths:
- /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logstash-tutorial.log

output.logstash:
  hosts: ["localhost:5044"] 

4.3、配置logstash.conf（输出日志到es）

这个是向es输入日志

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
} 

4.4、filebeat.xml配置多日志

配置四个日志文件进行采集：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2gekWe6B-1635087052708)(images/ksohtml/wpsFrHQOc.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-KOxRsArq-1635087052709)(images/ksohtml/wpsmek8Vx.jpg)]

配置：

type: log
enabled: true 
backoff: "1s"
tail_files: false 
paths:
- /Users/chyzhong/eclipse-study-workspace/11-springboottools/SpringBootLogback01/logs/SpringBootLogback01_*.log
fields:
filetype: SpringBootLogback01 # 加这个就是区分不同文件的
#fields:自定义字段
#ields_under_root 为true,则自定义字段将为文档中的顶级字段。
fields_under_root: true 

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QZrkbWlb-1635087052710)(images/ksohtml/wpsxASmgP.jpg)]

4.5、启动

$ cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/filebeat-7.9.3

$ ./filebeat

Exiting: error unpacking config data: can not convert 'string' into 'object' accessing 'output.logstash' (source:'filebeat.yml') accessing 'output' (source:'filebeat.yml')

$ nohup ./filebeat > logs/filebeat.log 2>&1 &

建议后台执行：

$ nohup ./filebeat -e -c filebeat.yml -d "publish" > logs/filebeat.log 2>&1 &

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BLAZUSQ6-1635087052710)(images/ksohtml/wpsaqTYr3.jpg)]

在logstash控制台打印如下信息：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Z120sj4H-1635087052711)(images/ksohtml/wpspzhFFB.jpg)]

4.6、启动文件（推荐）

sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/filebeat-7.9.3/start-filebeat.sh

4.7、查看进程

ps -ef|grep filebeat

4.8、查看日志（推荐）

tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/filebeat-7.9.3/logs/filebeat.log

elasticsearch更新数据：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ogdmkKvb-1635087052712)(images/ksohtml/wpsUGVWdg.jpg)]

filebeat更新数据：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Mvv3NNgP-1635087052712)(images/ksohtml/wps9ECTdW.jpg)]

Non-zero metrics in the last 30s

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-EjVvtgzj-1635087052713)(images/ksohtml/wpsyN5R7x.jpg)]

说明30s内无数据文件更新。但是日志文件是有更新的。

4.9、读取json格式日志

便于记录日志后，key成为索引，方便搜索。日志要提前设置成为json格式。

json.keys_under_root: true
json.overwrite_keys: true

- type: log
  enabled: true 
  backoff: "1s"
  tail_files: false 
  paths:
- /Users/chyzhong/eclipse-study-workspace/11-springboottools/SpringBootLogback01/logs/SpringBootLogback01_*.log
  fields:
  filetype: SpringBootLogback01 # 加这个就是区分不同文件的
  #fields:自定义字段
  #ields_under_root 为true,则自定义字段将为文档中的顶级字段。
  fields_under_root: true

改为：

-type: log
  enabled: true 
  backoff: "1s"
  tail_files: false 
  paths:
- /Users/chyzhong/eclipse-study-workspace/11-springboottools/SpringBootLogback01/logs/SpringBootLogback01_*.log
  fields:
  filetype: SpringBootLogback01 # 加这个就是区分不同文件的
  json.keys_under_root: true
  json.overwrite_keys: true

4、配置向控制台输出日志

保持first-pipeline.conf的配置

五、改变logstash输出

1、加入grok filter

first-pipeline.conf加入配置：

filter {
    grok {
       match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
}

filebeat停止，并删除data/registry，并重启，得到logstash控制台输出：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-7vEdRL4P-1635087052714)(images/ksohtml/wpsxnjIaS.jpg)]

2、用 Geoip 过滤器插件增强你的数据

first-pipeline.conf加入配置：

filter {
    grok {
       match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
    }
}

filebeat停止，并删除data/registry，并重启，得到logstash控制台输出：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-8qdv2SGD-1635087052715)(images/ksohtml/wpsNEb0OZ.jpg)]

3、索引你的数据到Elasticsearch

编辑first-pipeline.conf文件，替换output区域为：

output {
   #输出到控制台
   #stdout { codec => rubydebug }
   #输出到elasticsearch
   elasticsearch {
        hosts => [ "localhost:9200" ]
   }
}

4、查看elasticsearch索引（推荐）

$ curl 'localhost:9200/_cat/indices?v'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rpYXco5F-1635087052716)(images/ksohtml/wpskls2Oi.jpg)]

可以看到有一个名字叫"logstash-2018.08.11"的索引，其它的索引都是之前建的不用管

接下来，查看这个索引下的文档请求：

$ curl -X GET 'localhost:9200/logstash-2020.10.28-000001/_search?pretty&q=response=200'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cK9iyzlC-1635087052717)(images/ksohtml/wpsQDBnHs.jpg)]

$ curl -XGET 'localhost:9200/logstash-2020.10.28-000001/_search?pretty&q=geoip.city_name=Buffalo'

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gJPqUlQE-1635087052718)(images/ksohtml/wpsncnzhP.jpg)]

六、kibana展示日志

http://localhost:5601/

6.1、配置日志规则

点击顶部,弹出菜单,点击“Stack Management”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-VjB8zQoX-1635087052718)(images/ksohtml/wpsHXc8h5.jpg)]

点击“Index Patterns”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-mDqWcKHp-1635087052719)(images/ksohtml/wpsQ401zs.jpg)]

点击“Create index pattern”按钮

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-f8KlgITq-1635087052720)(images/ksohtml/wpsisPjTN.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-8JXcCV0i-1635087052721)(images/ksohtml/wpsUbC02L.jpg)]

填上过滤“logstash-2020.10.28*”，并点击“Next step”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xpAJrGPS-1635087052721)(images/ksohtml/wpsbyusGT.jpg)]

选择“@timestamp”, 点击“Create Index pattern”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-CcfVpbVx-1635087052722)(images/ksohtml/wps5i4W1x.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xFZSzLu7-1635087052723)(images/ksohtml/wpsP4yu8z.jpg)]

6.1、查看日志

点击“Discover”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-97FilCoS-1635087052724)(images/ksohtml/wps1buRwi.jpg)]

点击下拉框,选择对应的日志规则

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-eBVu2t0o-1635087052724)(images/ksohtml/wpswmmMHD.jpg)]

显示日志

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-22lDufGZ-1635087052727)(images/ksohtml/wpsdqJPWb.jpg)]

也可以根据日期控件选择时间,过滤日志:

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-uJv0Q7NG-1635087052728)(images/ksohtml/wpsFIA9Uf.jpg)]

6.2、设置显示日志

点击“Add”

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UxmVGgdH-1635087052730)(images/ksohtml/wps8JIM3Z.jpg)]

message就会在右边显示log文件的日志

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-XT8aJggm-1635087052732)(images/ksohtml/wpsDrFUJV.jpg)]

运行附录

# 启动应用
sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/start.sh
sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/start.sh
sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/start.sh
sh /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/filebeat-7.9.3/start-filebeat.sh
cd /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-head-master

# 启动grunt
grunt server

# 查看日志
tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/elasticsearch-7.9.3/logs/elasticsearch.log
tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/kibana-7.9.3/logs/kibana.log
tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/logstash-7.9.3-zip/logs/logstash.log
tail -f /Users/chyzhong/01-worktools/15-ELKF7.9.3-mac/filebeat-7.9.3/logs/filebeat.log

# 查看运行情况
ps -ef|grep filebeat
ps -ef|grep logstash
ps -ef|grep kibana
ps -ef|grep elasticsearch

 
filebeat少8小时

 # 启动nginx
sudo nginx -s reload