flume第四讲avro source(自带序列化机制)

flume第四讲avro source(自带序列化机制)

工作机制
通过监听一个网络端口来接收数据,而且接收的必须是avro序列化后的数据,
avro是序列化的框架是跨语言的
该source自己具备反序列化机制
一般用于agenet之间的级联
思想
我们从日志采集服务器上采集日志,采集的速度非常快,但是flume数据直接写入hdfs的速度非常慢,
时间已久就会出现数据积压的问题,我们的办法是将多设置集合agent,同时向hdfs上写入数据,环节积压压力
我们的avro就可以设置agent之间的级连

在虚拟就新建一个配置文件
[root@doit02 agent]# vi avro-m-1ogger.conf

[root@doit02 agent]# vi avro-m-1ogger.conf

a1.sources = s1
a1.channels = c1
a1.sinks = k1

a1.sources.s1.type = avro
a1.sources.s1.channels = c1
a1.sources.s1.bind = 0.0.0.0
a1.sources.s1.port = 8888

a1.channels.c1.type = memory

a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

启动avro的端口并输入数据

[root@doit02 flume-1.9.0-bin]# bin/flume-ng avro-client --host doit02 --port 8888
zaima
zaima
zaima
zaima
zaima

启动avro的配置文件

[root@doit02 flume-1.9.0-bin]# bin/flume-ng agent -n a1 -c conf -f \
agent/avro-m-1ogger.conf -Dflume.root.logger=INFO,console

监控端口回应(5个一批量)

2020-04-21 23:33:29,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 7A 61 69 6D 61                                  zaima }
2020-04-21 23:33:29,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 7A 61 69 6D 61                                  zaima }
2020-04-21 23:33:29,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 7A 61 69 6D 61                                  zaima }
2020-04-21 23:33:29,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 7A 61 69 6D 61                                  zaima }
2020-04-21 23:33:29,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 7A 61 69 6D 61                                  zaima }

拓展

我们后期会一到数据被压的问题,我们运用avro source的方式创建agent级联,就可以多个agent去写入hdfs,解决数据被压问题