flume作为日志实时采集的框架,可以与SparkStreaming实时处理框进行对接,flume实时产生数据,sparkStreaming做实时处理。Spark Streaming对接FlumeNG有两种方式,一种是FlumeNG将消息Push推给Spark Streaming,还有一种是Spark Streaming从flume 中Pull拉取数据。
实际使用中,使用pull方式较多,因为其使用到的SparkSink有缓冲作用,SparkStreaming通过有效的flumeReceiver从sink中拉取数据,拉取的数据有多副本存储方式,增加了容错性,稳定性好,并且在拉取过程中具有事务性,而push方式可能出现数据丢失。
- 安装flume1.6以上版本
- pom文件引入spark-streaming-flume-sink依赖
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume-sink_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
将spark-streaming-flume-sink_2.11.jar放在flume的lib目录
- 引入avro依赖:
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-ipc</artifactId>
<version>1.8.2</version>
</dependency>
将两jar包放在flume的lib目录,避免Could not initialize class rg.apache.spark.streaming.flume.sink.EventBatch
的错误出现
- pull方式:
配置文件 spark-streaming-flume-poll.conf:
b1.sources = r1
b1.sinks = k1
b1.channels = c1
#source
b1.sources.r1.type = netcat
b1.sources.r1.bind = localhost
b1.sources.r1.port = 44444
#channel
b1.channels.c1.type =memory
b1.channels.c1.capacity = 20000
b1.channels.c1.transactionCapacity=5000
#sinks
b1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
b1.sinks.k1.hostname=hadoop005
b1.sinks.k1.port = 7474
b1.sinks.k1.batchSize= 2000
#source-channel-sinks
b1.sources.r1.channels = c1
b1.sinks.k1.channel = c1
启动flume:
bin/flume-ng agent -n b1 -c conf -f job/spark-streaming-flume-poll.conf -Dflume.root.logger=INFO,console
启动telnet:
telnet localhost 44444
业务代码:
/**
* flume版wordcount
*/
object SparkPollFlume {
Logger.getLogger("org").setLevel(Level.ERROR)
def main(args: Array[String]): Unit = {
val conf: SparkConf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[*]")
val ssc = new StreamingContext(conf,Seconds(6))
val pollData: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createPollingStream(ssc,"hadoop005",7474)
pollData.map(x=>new String(x.event.getBody.array()).trim).flatMap(_.split(" +")).map((_,1)).reduceByKey(_+_).print(1000)
ssc.start()
ssc.awaitTermination()
}
}
//val flumeData: DStream[String] = pollData.map(x=> new String(x.event.getBody.array()).trim)
// val flatData: DStream[String] = flumeData.flatMap(_.split(" +"))
// val mapData: DStream[(String, Int)] = flatData.map((_,1))
// val result: DStream[(String, Int)] = mapData.reduceByKey(_+_)
// result.print(1000)
- push方式:
配置文件:
#push mode
b1.sources = r1
b1.sinks = k1
b1.channels = c1
#source
b1.sources.r1.type = netcat
b1.sources.r1.bind = localhost
b1.sources.r1.port = 44444
#channel
b1.channels.c1.type =memory
b1.channels.c1.capacity = 20000
b1.channels.c1.transactionCapacity=5000
#sinks
b1.sinks.k1.type = avro
b1.sinks.k1.hostname=192.168.100.121
b1.sinks.k1.port = 5555
b1.sinks.k1.batchSize= 2000
b1.sources.r1.channels = c1
b1.sinks.k1.channels = c1
业务代码:
object FlumePushSpark {
Logger.getLogger("org").setLevel(Level.ERROR)
def main(args: Array[String]): Unit = {
val conf: SparkConf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[*]")
val ssc = new StreamingContext(conf,Seconds(6))
val pollData: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createStream(ssc,"192.168.100.121",5555)
val originData: DStream[String] = pollData.map(x=>new String(x.event.getBody.array))
val flatMap: DStream[String] = originData.flatMap(_.split(" +"))
val wordWithOne: DStream[(String, Int)] = flatMap.map((_,1))
val result: DStream[(String, Int)] = wordWithOne.reduceByKey(_+_)
result.print(1000)
ssc.start()
ssc.awaitTermination()
}
}
注:
由于是push模式,需先将接受端程序端口打开(启动scala程序),再启动flume。
启动flume命令:
bin/flume-ng agent -n b1 -c conf -f job/spark-streaming-flume-push.conf -Dflume.root.logger=INFO,console
启动telnet:
telnet localhost 44444
报错:Could not configure sink k1 due to: No channel configured for sink: k1
,将配置文件的channels改为channel即可。