EFK之Fluentd收集日志到opensearch

Fluentd安装插件

由于 fluentd 不带 opensearch 插件，所以需要安装 fluent-plugin-opensearch 插件。
下载镜像

docker pull bitnami/fluentd:1.17.0

创建 dockerfile

FROM bitnami/fluentd:1.17.0 USER root
RUN gem install fluent-plugin-opensearch

构建一个新的镜像

docker build -t fluentd-elasticsearch .

启动命令

docker run -it -d -p 5140:5140 --name=fluentd -v
/opt/fluentd/fluent.conf:/opt/bitnami/fluentd/conf/fluentd.conf -v
/var/log/secure:/opt/bitnami/fluentd/secure -v
/var/log/command.log:/var/log/command.log fluentd-opensearch

注：secure 日志权限为 600，挂载到 docker 中无法读取，需要修改权限
chmod 604 /var/log/secure

Fluentd字段处理

Fluentd 需要配置输入日志，过滤规则，新增字段等，详细配置如下：

<source>
  @type tail
  path /opt/bitnami/fluentd/secure
  tag secure
  read_from_head false
  <parse>
	@type regexp
	expression /^(?<securetime>[^\]]*) (?<hostname>[^ ]*)  (?<sshd>sshd\[.*\]\:) (?<message>.*)$/
#	time_key securetime
#	time_format %Y-%m-%d %H:%M:%S
	keep_time_key true
  </parse>
</source>

<source>
  @type tail
  path /var/log/command.log
  tag command
  read_from_head false
  <parse>
	@type regexp
	expression 
/(?<cmdid>[^ ]*)\s\s(?<logtime>[^\]]*)\,(?<hosts>[^ ]*)\,(?<remoteip>[
 
^ ]*)\,(?<currentuser>[^ ]*)\,(?<user>[^ ]*)\,(?<cmd>.*)$/
	time_key logtime
	time_format %Y-%m-%d %H:%M:%S
	types cmdid:integer
	keep_time_key true
  </parse>

</source>

<filter command.**>
  @type record_transformer
  <record>
hostip 192. XXX.XXX.11	 typeID 2
  </record>
</filter>

<filter secure.**>
  @type record_transformer
  <record>
	hostip 192.XXX.XXX.11
	typeID 1
  </record>
</filter>

<match command.**>
  @type stdout
</match>
 
<match secure.**>
  @type opensearch
  host 172.XXX.XXX.115
  port 9200
  flush_interval 1s
  index_name secure_%Y.%m.%d
  logstash_format true
  logstash_dateformat %Y.%m.%d
  logstash_prefix secure
  target_index_affinity true
</match>

<match command.**>
  @type opensearch
	host 172.XXX.XXX.115
	port 9200
	flush_interval 1s
	index_name command_%Y.%m.%d
	logstash_format true
	logstash_dateformat %Y.%m.%d
	logstash_prefix command
	target_index_affinity true
</match>

输出到Opensearch
secure 日志

History 日志

通过验证，Fluentd实现将日志输出到opensearch中。