Flume+Kafka+Hdfs问题总结

最新推荐文章于 2023-11-17 00:45:07 发布

Jhon_yh

最新推荐文章于 2023-11-17 00:45:07 发布

阅读量442

点赞数

分类专栏： flume 文章标签： flume

flume 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

文章目录

数据漂移
文件丢失
问题描述

参考文章https://www.cnblogs.com/qiaoyihang/p/9229854.html

数据漂移

文件丢失

问题描述

2019-05-07 16:48:54,878 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://ns2/*/*/*/*/day_id=20190507/01-flumeEvents-2-.1557158424932.lzo_deflate.tmp
2019-05-07 16:54:09,357 (PollableSourceRunner-KafkaSource-r1) [INFO - org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:542)] Offset commit for group iisp_eda_iot_ddr_detail2Hive_group_new0 failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
2019-05-07 16:54:09,358 (PollableSourceRunner-KafkaSource-r1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:529)] Marking the coordinator 2147483643 dead.
2019-05-07 16:56:41,634 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:357)] Closing hdfs://ns2/*/*/*/*/day_id=20190507/01-flumeEvents-3-.1557158424932.lzo_deflate.tmp
2019-05-07 16:56:41,682 (hdfs-k3-call-runner-7) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:618)] Renaming hdfs://ns2/*/*/dwd_db.db/*/day_id=20190507/01-flumeEvents-3-.1557158424932.lzo_deflate.tmp to hdfs://ns2/*/*/*/*/day_id=20190507/01-flumeEvents-3-.1557158424932.lzo_deflate