【kafka】查看kafka消息队列文件

1.插件kafka消息队列文件 

[root@oracle1 kafka-0-logs]# cd sspu-nginx-0
[root@oracle1 sspu-nginx-0]# ll
total 608
-rw-rw-r-- 1 esadmin esadmin 10485760 Jul  1 11:12 00000000000000000000.index
-rw-rw-r-- 1 esadmin esadmin   608024 Jul  1 11:12 00000000000000000000.log
-rw-rw-r-- 1 esadmin esadmin 10485756 Jul  1 11:12 00000000000000000000.timeindex
-rw-rw-r-- 1 esadmin esadmin        8 Jul  1 11:12 leader-epoch-checkpoint
[root@oracle1 sspu-nginx-0]# strings 00000000000000000000.log |more
{"agent":{"id":"b22f72a4-463d-4370-8316-ca11475146f6","ephemeral_id":"9679aa80-4ff9-45e5-8096-62c27487af50"
,"name":"oracle1","type":"filebeat","version":"7.9.2","hostname":"oracle1"},"request_method":"GET","upstr_a
ddr":"-","@version":"1","status":403,"ups_resp_time":"-","domain":"localhost","host":{"name":"oracle1"},"by
tes":153,"request_time":"0.000","server_protocol":"HTTP/1.0","ecs":{"version":"1.5.0"},"referer":"-","input
":{"type":"log"},"request":"GET / HTTP/1.0","@timestamp":"2024-07-01T03:12:25.983Z","tags":["sspu-nginx","b
eats_input_codec_json_applied"],"upstr_host":"-","hostname":"oracle1","x_forwarded":"-","uri":"/index.html"
,"remote_addr":"192.168.1.7","http_host":"192.168.1.7","scheme":"http","log":{"offset":4825664,"file":{"pat
h":"/usr/local/nginx/logs/access-json.log"}}}
{"scheme":"http","request_method":"GET","agent":{"id":"b22f72a4-463d-4370-8316-ca11475146f6","ephemeral_id"
:"9679aa80-4ff9-45e5-8096-62c27487af50","name":"oracle1","type":"filebeat","version":"7.9.2","hostname":"or
acle1"},"@version":"1","status":403,"ups_resp_time":"-","domain":"localhost","host":{"name":"oracle1"},"byt
es":153,"request_time":"0.000","server_protocol":"HTTP/1.0","ecs":{"version":"1.5.0"},"referer":"-","reques
t":"GET / HTTP/1.0","input":{"type":"log"},"@timestamp":"2024-07-01T03:12:26.028Z","tags":["sspu-nginx","be
ats_input_codec_json_applied"],"upstr_host":"-","hostname":"oracle1","x_forwarded":"-","uri":"/index.html",
"remote_addr":"192.168.1.7","log":{"offset":5796260,"file":{"path":"/usr/local/nginx/logs/access-json.log"}
},"http_host":"192.168.1.7","upstr_addr":"-"}
{"agent":{"id":"b22f72a4-463d-4370-8316-ca11475146f6","ephemeral_id":"9679aa80-4ff9-45e5-8096-62c27487af50"
,"name":"oracle1","type":"filebeat","version":"7.9.2","hostname":"oracle1"},"request_method":"GET","upstr_a
ddr":"-","@version":"1","status":403,"ups_resp_time":"-","domain":"localhost","host":{"name":"oracle1"},"by
tes":153,"request_time":"0.000","server_protocol":"HTTP/1.0","ecs":{"version":"1.5.0"},"referer":"-","reque
st":"GET / HTTP/1.0","input":{"type":"log"},"@timestamp":"2024-07-01T03:12:26.028Z","tags":["sspu-nginx","b
eats_input_codec_json_applied"],"upstr_host":"-","hostname":"oracle1","x_forwarded":"-","uri":"/index.html"
,"remote_addr":"192.168.1.7","http_host":"192.168.1.7","scheme":"http","log":{"offset":5796733,"file":{"path":"/usr/local/nginx/logs/access-json.log"}}}

2.总结 

由此可见，kafka将数据直接接收，同时指定了:offset 偏移量。
而且kafka具有持久化的作用，可以缓存数据，并根据定期清除策略进行清理。

3.文件组成 

[root@oracle1 topic2-0]# ll
total 8
-rw-rw-r-- 1 esadmin esadmin 10485760 Jul  1 11:40 00000000000000000000.index
-rw-rw-r-- 1 esadmin esadmin     1076 Jul  1 11:48 00000000000000000000.log
-rw-rw-r-- 1 esadmin esadmin 10485756 Jul  1 11:40 00000000000000000000.timeindex
-rw-rw-r-- 1 esadmin esadmin        8 Jul  1 11:40 leader-epoch-checkpoint

每个topic由4种文件组成.log,存储具体接收的数据，index /timeindex 里面是空的。

leader-epoch-checkpoint leader的检查点信息。