flume输送数据到kafka报错（异机通信+Kafak本身配置问题）

最新推荐文章于 2022-08-02 09:21:17 发布
最新推荐文章于 2022-08-02 09:21:17 发布
阅读量1.7k 收藏 1
点赞数
文章标签： kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43390992/article/details/105316891
版权

1.对于配置文件，flume conf文件完全没有问题：
这里就不做粘贴了 （file-flume-kafka.conf）

2.检查flume日志文件，报错如下：

2019-05-17 09:38:27,185 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: k1 started
2019-05-17 09:39:00,287 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:00,289 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:35,314 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:39:35,314 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:40:10,345 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2019-05-17 09:40:10,346 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
        at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
        ... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired



ay.log, inode: 151415010, pos: 125835
2020-04-02 16:40:41,655 (PollableSourceRunner-TaildirSource-r1) [INFO - org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:283)] Opening file: /usr/local/logs/gateway/gateway.log, inode: 151415010, pos: 125835
2020-04-02 16:41:12,297 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2020-04-02 16:41:12,300 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
	... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2020-04-02 16:41:47,338 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:240)] Failed to publish events
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
2020-04-02 16:41:47,339 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
	at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:229)
	... 3 more

3、查找过程
（1）因为flume和kafka不在一台机器上，所以先看一下flume的端口是否开发

telnet kafkaIp 9092

我的flume没有安装telnet 尴尬，所以又安装了telnet和xinetd

https://www.cnblogs.com/zuochuang/p/6511285.html

安装好后测试 如下：
[root@localhost flume]# telnet 192.168.26.143 9092
Trying 192.168.26.143…
Connected to 192.168.26.143.
Escape character is ‘^]’.

OK 发现可以连接 说明端口已开通 flume这边应该是没有什么问题的，所以到kafka端查看。

（2）kafka的配置文件

百度找到的一个解决方法

https://www.cnblogs.com/EncryptingLife/p/7456188.html

我的kafka的配置文件内advertised.listeners=PLAINTEXT://ｉｐ:9092 没有配置
因为我的kafka和flume不在一台机器上 所以这个一定要配置

配置完后，重启kafaka就生效了！！！

具体的配置如下：
在这里插入图片描述
在这里插入图片描述

多学多总结，加油，伙伴们！！！

宝宝不哭^_^
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
flume+kafka+flink+mysql数据统计
04-20
本文将深入探讨这些技术及其在"flume+kafka+flink+mysql数据统计"中的应用。 首先，Flume 是 Apache Hadoop 的一个子项目，主要用于收集、聚合和移动大量日志数据。在本场景中，Flume 可以从 Nginx 服务器收集 Web ...
Flume采集数据到Kafka,然后从kafka取数据存储到HDFS的方法思路和完整步骤
03-08
自己研究大数据多年，写的一个日志数据采集方案笔记，可快速熟悉Flume，Kafka，Hdfs的操作使用，以及相互的操作接口。
flume往Kafka里写数据报错
迎接小天使
10-29 626
flume往Kafka里写数据报错： 04 Jul 2018 20:43:44,401 ERROR [kafka-producer-network-thread | producer-1] (org.apache.kafka.clients.producer.internals.Sender.run:130)  - Uncaught error in kafka producer I/O thr...
flume输送数据到kafka报错
weixin_43466575的博客
05-17 1295
1、flume的conf文件为： a1.sources = s1 a1.sinks = k1 a1.channels = c1 #source a1.sources.s1.type = exec a1.sources.s1.command=tail -F /root/teat/test.log a1.sources.s1.logStdErr=true # k1 a1.sinks.k1.type...
flume Kafka Channel 无法传输数据到kafka
weixin_43845733的博客
09-14 1347
文章目录第一个问题接下来紧接着就遇到第二个问题了接下来就来了第三个问题。 第一个问题 我使用的flume 1.5 但是现在官网上flume版本是1.9的，所以我把source定义为了 TailDir Source, 启动之后发现没有kafka中没有对应的topic, 然后参看flume启动日志，发现1.5版本没有TailDir Source, 我就把flume的版本升级到了1.7，TailD...
flume采集日志使用KafkaChannel传hdfs没反应解决方案
提桶跑路20年的博客
07-18 1733
学习到电商数仓的日志采集时，flume定义了filesource-KafkaChannel-无sink来生产日志放在kafka中，消费日志使用kafkaSource-fileChannel-hdfsSink来将日志放到hdfs中，好让日后进行相关计算业务。 问题：KafkaChannle读取了log文件，但是hdfsSink没有产生文件？ 方法： ①查看生产者flume日志，和消费者flume日志，看有没有相关ERROR 启动生产者和消费者的命令应该加上写到文件中，不然无法看出错误 “nohup /home
学习hadoop遇到的问题（flume-kafka报错）
小马哥的博客
04-16 2327
org.apache.kafka.common.errors.InterruptException: Flush interrupted. at org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:546) at org.apache.flume.sink.kafka.KafkaSink.proce...
基于Flume+Kafka+Hbase+Flink+FineBI的实时综合案例.txt
最新发布
10-13
基于Flume+Kafka+Hbase+Flink+FineBI的实时综合案例.txt基于Flume+Kafka+Hbase+Flink+FineBI的实时综合案例.txt基于Flume+Kafka+Hbase+Flink+FineBI的实时综合案例.txt基于Flume+Kafka+Hbase+Flink+FineBI的实时综合...
flume-ng+Kafka+Storm+HDFS实时系统组合
01-30
5.使用什么脚本可以查看flume有没有往Kafka传输数据做软件开发的都知道模块化思想，这样设计的原因有两方面：一方面是可以模块化，功能划分更加清晰，从“数据采集--数据接入--流失计算--数据输出
Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+ES+Redash等详细安装部署
06-22
1、内容概要：Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南（详细搭建步骤+实践过程问题总结）。 2、适合人群：大数据运维、大数据相关技术及组件初学者。 3、...
flume和kafka合成报错ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleS
小蚯蚓的博客
09-23 4782
0.环境： flume1.6.0和kafka_2.10-0.8.2.1 1.问题描述： flume和kafka集成，flume输出到kafka： (1)flume配置文件： agent1.sources=avro-source agent1.channels=logger-channel agent1.sinks=kafka-sink #define source agent1...
Flume Sink到Kafka遇到的问题
博客
12-02 4135
1、版本问题 2018-12-02 15:23:06,334 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:427)] Sink k1 has been removed due to an e...
Flume连接Kafka的broker出错
weixin_30420305的博客
08-30 230
在启动Flume的时候，出现下面的异常，但是程序照样能运行，Kafka也能够收到数据，只是偶尔会断点。 2016-08-25 15:32:54,561 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Fetching metadata...
解决flume采集日志使用KafkaChannel写不到hdfs的问题
JiaXingNashishua的博客
08-02 2025
重点，我就是这里出错了）然后将这个jar包放到hadoop104的/opt/module/flume/lib文件夹下面，然后分发给hadoop102,hadoop103中。检查一下消费flume所在的集群，也就是hadoop104中的/opt/module/flume/conf目录下的kafka-flume-hdfs.conf文件。附（抛出一下异常，说明你的消费flume内存不够，只要修改配置就可以了）这里，大家的包名可能不一样，根据自己创建的包名进行修改即可。最后，重新启动一下集群，就可以了。...
flume-使用KafkaChannel读取不到数据
epitomizelu的专栏
03-25 2997
使用TAILDIR监听日志写入KafkaChannel。 配置如下： a1.sources = r1 a1.channels = c1 #描述source a1.sources.r1.type = TAILDIR a1.sources.r1.filegroups = f1 a1.sources.r1.filegroups.f1 = /opt/module/applog/log22/app.* a1.sources.r1.positionFile = /opt/module/flume/taildir_po
Kafka报错：org.apache.flume.ChannelException: Take failed due to IO error [channel=c1]
Silver_Sword的博客
08-09 2338
2018-08-09 11:32:06,186 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.file.LogFile$RandomReader.checkOut(LogFile.java:560)] Opening /home/bigdata/flume/data/log-2 fo...
（file-flume-kafka-flume-hdfs过程中）flume 配置文件的编写
Huc673619的博客
10-17 550
flume 配置文件的编写 flume 配置文件 file-flume-kafka.conf 使用 TAILDIR source 1 # Name the components on this agent 2 a1.sources = r1 3 a1.channels = c1 4 5 # Describe/configure the source 6 a1.sources.r1.type = TAILDIR 7 a1.sources.r1.filegroups = f1
kafka无法收到flume采集的数据的解决办法
little prince，blue coder
07-23 7611
问题重现 在写黑名单那篇博文的时候，我是通过直接copy log日志文件到监控目录下的方式来模拟数据的，在前几次模拟访问日志文件的时候挺正常的，copy进去基本都是秒采集（文件显示直接加了.COMPLETED后缀）。 但到后来再往采集目录下copy log日志文件的时候，待采集目录下的文件并不会显示被采集（文件没有.COMPLETED后缀），kafka也一直收不到flume采集来的数据。但重启...
kafka的配置文件说明与flume整合kafka
bitmao888的博客
05-15 566
文章目录配置文件说明Server.properties配置文件说明producer生产者配置文件说明consumer消费者配置详细说明flume整合kafka创建文件夹配置flume_kafka.conf启动flume启动kafka消费把文件放进/export/servers/flumedata，在kafka即可看到数据 配置文件说明 Server.properties配置文件说明 #broker的全局唯一编号，不能重复 broker.id=0 #用来监听链接的端口，producer或consumer将在
flume数据采集到kafka
08-23
Flume是一个可以将数据从不同的源头（如日志文件、网络、消息队列等）采集到目标系统（如HDFS、Hbase等）的工具。其中，Flume可以将数据采集到Kafka作为中间存储。数据从源头经过Flume节点采集后，通过KafkaSink将数据发送到Kafka集群中的指定Topic中，以供后续的处理和存储。这样可以实现高效且稳定的数据采集。通过配置Flume的sink，指定Kafka的broker地址、端口号、Topic等参数，可以将数据成功地采集到Kafka。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* *2* *3* [数据采集组件：Flume基础用法和Kafka集成](https://blog.csdn.net/cicada_smile/article/details/114380191)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 100%"] [ .reference_list ]

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值