sparkStreaming整合flume

sparkStreaming整合flume

• 需要把spark-streaming-flume-sink_2.11-2.1.3.jar放到flume的lib目录下
• 使用scala-library-2.11.8.jar替换之前的scala-library-2.10依赖的jar包

9.1 poll拉模式整合

• 1、flume配置文件

        a1.sources = r1
        a1.sinks = k1
        a1.channels = c1
        
        #source
        a1.sources.r1.channels = c1
        a1.sources.r1.type = spooldir
        a1.sources.r1.spoolDir = /root/data
        a1.sources.r1.fileHeader = true
        
        #channel
        a1.channels.c1.type =memory
        a1.channels.c1.capacity = 20000
        a1.channels.c1.transactionCapacity=5000
        
        #sinks
        a1.sinks.k1.channel = c1
        a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
        a1.sinks.k1.hostname=yh
        a1.sinks.k1.port = 8888

下载地址
 - http://spark.apache.org/docs/latest/streaming-flume-integration.html
 - 下载位置
![图片描述][1]