flume输送数据到kafka报错

最新推荐文章于 2022-12-16 18:02:52 发布
最新推荐文章于 2022-12-16 18:02:52 发布
阅读量1.2k 收藏
点赞数
文章标签： flume kafka 报错
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43466575/article/details/90288460
版权

1、flume的conf文件为：

a1.sources = s1
a1.sinks = k1
a1.channels = c1

#source
a1.sources.s1.type = exec
a1.sources.s1.command=tail -F /root/teat/test.log
a1.sources.s1.logStdErr=true

# k1
a1.sinks.k1.type= org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList=10.0.0.214:9092
a1.sinks.k1.topic=ADanalysis
a1.sinks.k1.serializer.class=kafka.serializer.StringEncoder

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 100000

a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

2、flume日志报错内容：

2019-05-17 09:38:27,185 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: k1 started
2019-05-17 09:39:00,287 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:00,289 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:35,314 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:35,314 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:40:10,345 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:40:10,346 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired

3、查找过程
（1）因为flume和kafka不在一台机器上，所以先看一下flume的端口是否开发

telnet kafkaIp 9092

我的flume没有安装telnet 尴尬，所以又安装了telnet和xinetd

https://www.cnblogs.com/zuochuang/p/6511285.html

OK 发现可以连接 说明端口已开通 flume这边应该是没有什么问题的，所以到kafka端查看。

（2）kafka的配置文件

百度找到的一个解决方法

https://www.cnblogs.com/EncryptingLife/p/7456188.html

我的kafka的配置文件内advertised.listeners=PLAINTEXT://ｉｐ:9092 没有配置
因为我的kafka和flume不在一台机器上 所以这个一定要配置

mmaqzxc
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
Flume+Kafka消费实时日志
Renxiao的专栏
12-20 3306
环境安装 前提是已经安装好JDK1.8 Hadoop2.7  zookeeper3.4 scala2.12  1安装Kafka 1)下载安装包 参考：http://blog.csdn.net/u014035172/article/details/68061463 首先，官网上下载最新版本的Kafka，解压到某个文件夹 2)配置环境：编辑里面的server.properties文件，主要...
Flume对接kafka
m0_58342743的博客
11-07 572
1.简单的对接 需求： 使用收集控制台的信息存储再kafka并用kafka消费者消费 Flume配置文件： # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = 0.0.0.0 a1.sources.r1.port = 444
flume往Kafka里写数据报错
迎接小天使
10-29 631
flume往Kafka里写数据报错： 04 Jul 2018 20:43:44,401 ERROR [kafka-producer-network-thread | producer-1] (org.apache.kafka.clients.producer.internals.Sender.run:130)  - Uncaught error in kafka producer I/O thr...
云主机：flume 配置连接kafka（踩坑 3 记：flume无法将监控数据写入kafka）
leofionn的博客
05-04 1100
1.Jar放进lib目录 [root@yws76 lib]# ll total 51064 -rw-r--r-- 1 root root 23819 Jan 12 11:23 AdvancedExecSource.jar 2.配置文件： [root@yws76 apache-flume-1.7.0-bin]# cat conf/exec_memory_kafka.properties # N...
Failed to send data to Kafka
malove110的博客
12-16 1621
kafka 异常
flume输送数据到kafka报错（异机通信+Kafak本身配置问题）
weixin_43390992的博客
04-04 1746
1.对于配置文件，flume conf文件完全没有问题： 这里就不做粘贴了 （file-flume-kafka.conf） 2.检查flume日志文件，报错如下： 2019-05-17 09:38:27,185 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.star...
flume+kafka+flink+mysql数据统计
04-20
接着，Kafka 是一个高吞吐量的分布式消息队列系统，它在 Flume 收集到数据后，作为一个中间层存储平台，能够缓存大量数据并确保数据的可靠传输。Kafka 的主题和分区机制保证了数据的有序性和容错性，使得即使在高...
Flume 抽取MYSQL Oracle数据 JSON格式 推送Kafka
05-28
Flume二次开发，支持抽取MYSQL Oracle数据库数据 以JSON格式推送至Kafka。 demo： sql_json.sources.sql_source.type = com.hbn.rdb.source.SQLSource sql_json.sources.sql_source.connectionurl = jdbc:oracle:...
Kafka hdfs flume 数据采集实验
最新发布
11-17
Flume 提供了简单灵活的架构，支持多个数据源（如Web服务器日志、Kafka），通过数据流将数据传输到一个或多个目的地，如HDFS。在这个实验中，Flume 可能被配置为Kafka消费者，从Kafka中读取数据，并将其存储到HDFS中...
kafka+flume 实时采集oracle数据到hive中.docx
08-11
基于Kafka+Flume实时采集Oracle数据到Hive中 一、Kafka获取Oracle日志实时数据 Kafka是一种分布式流媒体平台，能够实时地从Oracle数据库中提取日志信息。为了实现这一点，需要先安装ZooKeeper和Kafka，然后配置...
flume数据丢失与重复_Kafka精通：Producer发送数据丢失、重复、乱序原因及解决方案...
weixin_36459720的博客
01-01 962
生产环境对于生产者来说，Kafka集群发送消息经常会遇到消息丢失、重复、乱序等问题，下面我们来讲解一下出现这些问题的原因及解决方案。1.我们知道Kafka为保障数据的可靠性，采用了多副本的存储机制假设一个Topic拆分为了3个Partition，分别是PartitionA，PartitonB，PartitionC，此时每个Partition都有2个副本。比如Partiti...
flume采集数据采集与kafaka交互报错：java.lang.OutOfMemoryError:java heap space
weixin_43390992的博客
04-07 1193
详细报错如下图： ##解决办法： 进入flume bin目录下，修改flume-ng文件，如下图： 红框的地方默认是20m，可以根据自己服务器内存大小设置，我这里设置的是10240m(也就是10个g) 根据以上的报错，自己也进行搜索并整理了细节的东西，供需要的小伙伴们参考： 博文地址： 记flume部署过程中遇到的问题以及解决方法 配置flume采集kafaka日志导入hdfs（采用的file...
Failed to publish events kafka.common.FailedToSendMessageException: Failed to send messages after 3
weixin_41666051的博客
01-08 2405
flume监听数据发送给kafka的消费者，出现情况，kafka重复消费flume的数据，查看了flume的日志报错如下 (org.apache.flume.sink.kafka.KafkaSink.process:145) - Failed to publish events kafka.common.FailedToSendMessageException: Failed to send m...
flume接收kafka消息过多导致的GC错误的解决办法
sxj的专栏
04-22 2703
当flume接收kafka消息过多会导致如下错误： Exception in thread "PollableSourceRunner-KafkaSource-s1" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.StringCoding$StringDecoder.decode(StringCodin...
解决Flume数据采集中出现的几个问题
ForestCat的专栏
06-23 1573
22/06/23 12:08:58 ERROR hdfs.HDFSEventSink: process failed sinkjava.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
flume读取kafka写入hive异常解决-Unable to deliver event. Exception follows.
hello_java_lcl的博客
05-18 2450
用flume访问kafka，写入hive数据库时，启动命令：bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console 配置文件是安装flume官方文档配置的，很全，地址：http://flume.apache.org/FlumeUserGuide.html#kafka-sink 启动flume之后，报错： ERROR flume.SinkRunner: Unable t
flume对接kafka
weixin_43003792的博客
02-01 222
需求 flume采集的数据发送到不同的kafka topic里面。 flume-kafka.conf define a1.sources = r1 a1.sinks = k1 a1.channels = c1 source a1.sources.r1.type = exec a1.sources.r1.command = tail -F -c +0 /opt/module/datas/flume.log a1.sources.r1.shell = /bin/bash -c sink a1.sinks.k1.
【硬刚Kafka】KAFKA基础（十五）：Flume对接Kafka
微信搜：import_bigdata，大数据领域硬核原创作者
09-23 448
本文是对《【硬刚大数据之学习路线篇】从零到大数据专家的学习指南(全面升级版)》的Kafka：部分补充。 1简单实现 1）配置flume # define a1.sources = r1 a1.sinks = k1 a1.channels = c1 # source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/module/data/flume.log # sink a1.sinks.k1.type = org..
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值