Apache Flume是一个用来有效地收集,聚集和移动大量日志数据的分布式的,有效的服务。这里我们解释一下怎样配置Flume和Spark Streaming来从Flume获取数据,我们让flume直接将收集到的数据发送到spark streaming去处理
package com.pinganfu.flumespark
import org.apache.spark.SparkConf
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.flume.FlumeUtils
import org.apache.spark.streaming.{Milliseconds, StreamingContext}
object SparkFlumeEvent{
def main(args: Array[String]) {
val batchInterval = Milliseconds(5000)
val sparkConf = new SparkConf().setAppName("flumetospark").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, batchInterval)
val stream = FlumeUtils.createStream(ssc,"localhost",33333, StorageLevel.MEMORY_AND_DISK)
stream.count().map(cnt => "Received " + cnt + " flume events." ).print()
ssc.start()
ssc.awaitTermination()
}
}
Flume配置
a1.channels = c1
a1.sinks = k1
a1.sources = r1
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = localhost
a1.sinks.k1.port = 33333
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
a1.sources.r1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
启动三个窗口
先启动spark-shell(引入两个依赖包)
bin/spark-shell --jars lib/spark-streaming-flume_2.10-1.0.0.jar,lib/flume-ng-sdk-1.6.0.jar
然后将上面代码贴进去
再启动flume
bin/flume-ng agent --conf conf --conf-file conf/flume.conf --name a1 -Dflume.root.logger=INFO,console
再启动telnet localhost 44444
发数据
截图如下: