![](https://img-blog.csdnimg.cn/20200122204322106.png?x-oss-process=image/resize,m_fixed,h_224,w_224)
Flume
Flume安装;内部组成(source,channel,sink);拦截器;报错及解决方法
struggle@徐磊
知识重要,态度更重要!
展开
-
Error while getting events from Kafka
Channel的type类型是Kafka结果报错错误详情: Error while getting events from Kafka解决方法:最后加一行 a1.channels.c1.parseAsFlumeEvent=false原创 2020-03-03 21:42:01 · 571 阅读 · 0 评论 -
保证Flume数据不丢失
1、断点续传注意 : flume1.7之后有,1.7之前就需要自己写jar包上传上去使用#source的配置# source类型a1.sources.r1.type = TAILDIR# 元数据位置a1.sources.r1.positionFile = /home/hadoop/data/bd/taildir_position.json# 监控的目录a1.sources.r1.f...原创 2020-02-24 21:27:12 · 2628 阅读 · 0 评论 -
flume往hbase上传报错: ERROR hbase.HBaseSink: Failed to commit transaction.Transaction rolled back.
错误:20/02/02 12:18:35 ERROR hbase.HBaseSink: Failed to commit transaction.Transaction rolled back.java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V at org.apac...原创 2020-02-02 13:24:41 · 949 阅读 · 0 评论 -
Flume拦截器、Process、多路复用的使用
#flume扇出的时候的多路复用形式a1.sources=r1a1.channels=c1 c2a1.sinks=s1 s2a1.sources.r1.type=HTTPa1.sources.r1.port=4444a1.sources.r1.selector.type=multiplexinga1.sources.r1.selector.header=a# 是a1的话就发送到...原创 2020-01-23 14:11:01 · 331 阅读 · 0 评论 -
flume中的sink往hdfs上寫的時候,報錯WARN hdfs.BucketWriter: failed to rename() file (hdfs://node132/flume/FlumeD
报错:用命令启动:bin/flume-ngagent--conf./conf/-fconf/flume-conf.conf-nagent1一切正常,而且flume.log日志也正常。用bin/flume-ngavro-client-Hkf-app1-p44444-F/chunk1/demo/flume/test2.txt发送文件,flume.log如下:[d...原创 2020-01-22 20:48:23 · 642 阅读 · 0 评论 -
Flume的source、channel,sink简单介绍及使用
Source# source-spooldir 目錄一經監聽就不能再做修改,也不能再重命名a1.sources=r1a1.channels=c1a1.sinks=s1a1.sources.r1.type=spooldira1.sources.r1.spoolDir=/root/data/testa1.channels.c1.type=memorya1.channel...原创 2020-01-22 20:40:18 · 2884 阅读 · 0 评论