Flume中使用KafkaChannel时topic中存在异常奇怪字符

Kafka的topic中每一行前面都会有奇怪的字符，方框或者点或者问号等等，如图：

 ⬜I am row one.
  I am row two.
  I am row three.
 &I am the other row.

解决方案：

将flume中的配置文件.conf文件中的kafkaChannel的属性parseAsFlumeEvent默认为true，改为false即可。

如下：其中a1为agent的名称，c1为channel名称

a1.channels.c1.parseAsFlumeEvent = false

 

测试结果如下：^*^ OK啦！

I am row one.
I am row two.
I am row three.
I am the other row.

原因：

Flume官网对该属性的解释是：

 

Property Name	Default	Description
parseAsFlumeEvent	true	Expecting Avro datums with FlumeEvent schema in the channel. This should be true if Flume source is writing to the channel and false if other producers are writing into the topic that the channel is using. Flume source messages to Kafka can be parsed outside of Flume by using org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact

 ^0^不翻译了，反正我也看不懂

 

该属性为true时会转成Flume Event，结果时把Flume的headers中的信息混合着内容一起写入kafka的消息中，无论headers是否为空时都出现了奇怪的特殊字符（已测试）。

 

但是，该属性设置为false时，有些版本可能不起作用，因为这曾是一个BUG，后来修复了但不知是哪个版本之后了（懒得查），目前我用过的1.5.2和1.7.0都可以。 Flume的1.7.0版本改进的列表包括：

[FLUME-2781] - A KafkaChannel defined as parseAsFlumeEvent=false cannot be correctly used by a Flumesource

上面内容的网址连接：https://issues.apache.org/jira/browse/FLUME-2781

 

主要内容：Implemented a tiny change so putting to the channel isconsistent with take, so if parseAsFlumeEvent is false it uses the event bodytext instead of an Avro Flume event object.

放入channel的内容与取时保持一致。theevents are written as text and not as an Avro object.