SparkStreaming基于Kafka Receiver

1.流程图

 

2.streaming代码

 

SparkConf conf = new SparkConf().setMaster("local[4]").setAppName("SparkStreamingOnKafkaRecevier");

 

JavaStreamingContext jsc = new JavaStreamingContext(conf, Durations.seconds(30));

Map<String, Integer> topic = new HashMap<String, Integer>();

topic.put("SparkStreamingOnKafkaRecevier", 2);

JavaPairReceiverInputDStream<String, String> lines = KafkaUtils.createStream(jsc, "rizhicaiji:2181", "MyFirstGroup", topic);

 

JavaDStream<String> words = lines.flatMap(new FlatMapFunction<Tuple2<String,String>, String>() { //如果是Scala，由于SAM转换，所以可以写成val words = lines.flatMap { line => line.split(" ")}

 

public Iterable<String> call(Tuple2<String,String> tuple) throws Exception {

return Arrays.asList(tuple._2.split(" "));

}

});