suojie的博客

异常如下：19/01/18 02:12:51 WARN hive.HiveSink: k1 : Failed connecting to EndPoint {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] }org.apache.flume.sink.hive...

1.一个简单的例子1.写配置文件01localDir_local.flu#flume中一个基本单元式agent，可以有多个channel,多个sink#给代理起名称， 不需要声明直接使用#分布给a1代理下的source channel sink起一个名称#a1代理下sources有一个名为s1#如果有多个：a1.channels=c1 c2 c3sources = s1a1....

a1.sources = s1a1.channels = c1a1.sinks = k1a1.sources.s1.type = avroa1.sources.s1.channels = c1a1.sources.s1.bind = wangfutaia1.sources.s1.port = 4141a1.channels.c1.type = filea1.channels....

同一时刻，source只能传输给一个channelsource是通过event header来决定传输到哪一个channelsource 本地channel c1 localfile c2 memsink k1 logger k2本地目录如果文件名1_txt 传输到logger 如果是2_txt 传输到本地a1.sources = s1 a1.channels =...

a1.sources = s1a1.channels=c1 c2a1.sinks=k1 k2a1.sources.s1.type = syslogtcpa1.sources.s1.port = 5140a1.sources.s1.host = wangfutaia1.sources.s1.channels = c1 c2配置多路复制a1.sources.s1.sel...

a1.sources = s1a1.sinks = k1a1.channels = c1a1.sources.s1.type = httpa1.sources.s1.port = 5140a1.sources.s1.bind = 192.168.15.76a1.sources.s1.channels = c1a1.sources.s1.handler = org.apache.fl...

a1.sources = s1a1.channels = c1a1.sinks = k1#tcp协议a1.sources.s1.type = syslogtcpa1.sources.s1.port = 5140a1.sources.s1.host = wangfutaia1.sources.s1.channels = c1#channela1.channels.c1.typ...

方案一：收集到hdfs中方案二：插入已经有的表，使用flume收集数据到hive，hive中数据必须以orc格式保存source 网络日志channel 本地磁盘+memory，优先使用内存，如果内存使用完毕，就使用本地磁盘作为缓冲sink hivea1.sources = s1a1.channels=c1a1.sinks=k1#tcp协议a1.sources.s1.t...

例子：a1.sources = s1a1.channels=c1a1.sinks=k1#tcp协议a1.sources.s1.type = syslogtcpa1.sources.s1.port= 5140a1.sources.s1.host= wangfutaia1.sources.s1.channels = c1a1.channels = c1a1.channels....

#channels  内存#sink 本地a1.sources = s1a1.channels=c1a1.sinks=k1#tcp协议#a1.sources.s1.type =  syslogtcp#a1.sources.s1.port= 5140#a1.sources.s1.host= wangfutai#a1.sources.s1.channels = c1...

a1.sources = s1a1.channels=c1a1.sinks=k1 a1.sources.s1.type =spooldira1.sources.s1.channels=c1a1.sources.s1.spoolDir=/home/wang/a/flume/logsa1.sources.s1.fileHeader=truea1.channels = c1a1.ch...