改造org.apache.flume.source.kafka.KafkaSource类

诉求：系统通过kafka对接上游数据，上游数据格式使用json封装，因flume自带的KafkaSource类不能做到将kafka接收消息时间加入到落地消息中。所以有本次改造需求：获取数据进入kafka的时间，并将此时间放入json数据中。参考博客资料有flume采集kafka数据自定义source_自定义kafkasource_狮子王K的博客-CSDN博客

过程遇到的问题有：

1、java.lang.NoSuchMethodError：org.apache.flume.Context.getSubProperties(Ljava/lang/String;)Lcom/google/common/collect/ImmutableMap

错误原因是开源社区的插件是apache flume版本的，与CDH版本的不兼容，引用对应CDH的flume版本即可解决

2、本地maven打包时报错：Failure to find com.sun.jdmk:jmxtools:jar:1.2.1，出现缺失com.sun.jdmk:jmxtools:jar:1.2.1的错误提示。解决如下，引入一下dependency

<dependency>  
            <groupId>log4j</groupId>  
            <artifactId>log4j</artifactId>  
            <version>1.2.15</version>  
            <exclusions>  
                <exclusion>  
                    <groupId>javax.jms</groupId>  
                    <artifactId>jms</artifactId>  
                </exclusion>  
                <exclusion>  
                    <groupId>com.sun.jdmk</groupId>  
                    <artifactId>jmxtools</artifactId>  
                </exclusion>  
                <exclusion>  
                    <groupId>com.sun.jmx</groupId>  
                    <artifactId>jmxri</artifactId>  
                </exclusion>  
            </exclusions>  
</dependency>

关键代码展示

        // get next message
        ConsumerRecord<String, byte[]> message = it.next();

        //log.info("message的内容是：" + message.toString());
        //log.info("message.value的内容是：" + new String(message.value(),"UTF-8"));
        String value = new String(message.value(),"UTF-8");
        JSONObject jsonObject = JSON.parseObject(JSONObject.parse(value).toString());
        //log.info("jsonObject的内容是：" + jsonObject);


        //获取record中的timestamp
        timestamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(message.timestamp());

        //将kafka接收数据时间写入到json消息中
        jsonObject.put("kafkaTime", timestamp);
        String kafkaTimeMessage = jsonObject.toString();
        log.info("kafkaTimeMessage的内容是：" + kafkaTimeMessage);


        kafkaKey = message.key();

        if (useAvroEventFormat) {
          //Assume the event is in Avro format using the AvroFlumeEvent schema
          //Will need to catch the exception if it is not
          ByteArrayInputStream in =
                  new ByteArrayInputStream(kafkaTimeMessage.getBytes());
          decoder = DecoderFactory.get().directBinaryDecoder(in, decoder);
          if (!reader.isPresent()) {
            reader = Optional.of(
                    new SpecificDatumReader<AvroFlumeEvent>(AvroFlumeEvent.class));
          }
          //This may throw an exception but it will be caught by the
          //exception handler below and logged at error
          AvroFlumeEvent avroevent = reader.get().read(null, decoder);

          eventBody = avroevent.getBody().array();
          headers = toStringMap(avroevent.getHeaders());
        } else {
          eventBody = kafkaTimeMessage.getBytes();
          headers.clear();
          headers = new HashMap<String, String>(4);
        }