Spark Streaming 实例：滑动窗口Window中的数据进行累加 —— 读取Kafka数据实现最近一小时能耗数据统计

数据格式如下：电表标签+每15min的用电量（度）

MT_001:0;MT_002:5;MT_003:0;MT_004:40;MT_005:20;
MT_001:0;MT_002:5;MT_003:0;MT_004:30;MT_005:15;
MT_001:0;MT_002:4;MT_003:0;MT_004:30;MT_005:13;
MT_001:0;MT_002:4;MT_003:0;MT_004:31;MT_005:14;
MT_001:0;MT_002:4;MT_003:0;MT_004:29;MT_005:13;

实际数据是有370个电表，一个月的数据采样周期15min，可以到这里下载：https://download.csdn.net/download/Ahuuua/11996754

 

目标：统计每个电表最近一小时的耗电量

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.*;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaInputDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka010.ConsumerStrategies;
import org.apache.spark.streaming.kafka010.KafkaUtils;
import org.apache.spark.streaming.kafka010.LocationStrategies;
import scala.Tuple2;

import java.util.*;

public class SparkStreamingConsumer {
    public static void main(String[] args) throws InterruptedException {
        SparkConf conf = new SparkConf();
        conf.setAppName("kafkaSpark");
        conf.setMaster("local[4]");
        //创建Spark流应用上下文，batch设为15min
        JavaStreamingContext streamingContext = new JavaStreamingContext(conf,new Duration(15*60*1000));
        Map<String, Object> kafkaParams = new HashMap<>();
        kafkaParams.put("bootstrap.servers", "b2:9092,b3:9092");
        kafkaParams.put("key.deserializer", StringDeserializer.class);
        kafkaParams.put("value.deserializer", StringDeserializer.class);
        kafkaParams.put("group.id", "g6");
        kafkaParams.put("auto.offset.reset", "latest");
        kafkaParams.put("enable.auto.commit", false);

        Collection<String> topics = Arrays.asList("energylogforsparkstreaming");

        final JavaInputDStream<ConsumerRecord<String, String>> stream =
                KafkaUtils.createDirectStream(
                        streamingContext,
                        LocationStrategies.PreferConsistent(),
                        ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)
                );

        //压扁
        JavaDStream<String> wordsDS = stream.flatMap(new FlatMapFunction<ConsumerRecord<String,String>, String>() {
            public Iterator<String> call(ConsumerRecord<String, String> r) throws Exception {
                String value = r.value();
                List<String> list = new ArrayList<String>();
                String[] arr = value.split(";");
                for (String s : arr) {
                    list.add(s);
                }
                return list.iterator();
            }
        });

        //映射成元组，（电表名称，近15min用电量）
        JavaPairDStream<String,Integer> pairDS = wordsDS.mapToPair(new PairFunction<String, String, Integer>() {
            public Tuple2<String, Integer> call(String s) throws Exception {
                //电表：用电量
                String[] arr=s.split(":");
                return new Tuple2<String,Integer>(arr[0],Integer.parseInt(arr[1]));
            }
        }) ;

        //聚合reduceByKeyAndWindow（电表名称，近一小时用电量），窗口长度设为1小时，滑动间隔设为15min
        JavaPairDStream<String,Integer> countDS = pairDS.reduceByKeyAndWindow(new Function2<Integer, Integer, Integer>() {
            public Integer call(Integer v1, Integer v2) throws Exception {
                return v1 + v2;
            }
        },new Duration(60*60*1000),new Duration(15*60*1000));
        //打印
        countDS.print();

        streamingContext.start();
        streamingContext.awaitTermination();
        streamingContext.stop();
    }
}

依赖如下（有些不需要）：

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.17</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>
    </dependencies>

测试：

1.  缩小时间间隔：

     批次batch设为了new Duration(2*1000)，窗口和滑动间隔设为了new Duration(16*1000),new    Duration(6*1000)

2.  开一个kafka生产者

     kafka-console-producer.sh --broker-list b2:9092 --topic energylogforsparkstreaming

3. 开启测试程序，kafka生产数据

 

$ kafka-console-producer.sh --broker-list b2:9092 --topic energylogforsparkstreaming
MT_001:0;MT_002:5;MT_003:0;MT_004:40;MT_005:20;
MT_001:0;MT_002:5;MT_003:0;MT_004:30;MT_005:15;
MT_001:0;MT_002:4;MT_003:0;MT_004:30;MT_005:13;
MT_001:0;MT_002:4;MT_003:0;MT_004:31;MT_005:14;

输出如下：

-------------------------------------------
Time: 1574670736000 ms
-------------------------------------------

-------------------------------------------
Time: 1574670742000 ms
-------------------------------------------
(MT_004,40)
(MT_001,0)
(MT_005,20)
(MT_002,5)
(MT_003,0)

-------------------------------------------
Time: 1574670748000 ms
-------------------------------------------
(MT_004,70)
(MT_001,0)
(MT_005,35)
(MT_002,10)
(MT_003,0)

-------------------------------------------
Time: 1574670754000 ms
-------------------------------------------
(MT_004,60)
(MT_001,0)
(MT_005,28)
(MT_002,9)
(MT_003,0)

-------------------------------------------
Time: 1574670760000 ms
-------------------------------------------
(MT_004,61)
(MT_001,0)
(MT_005,27)
(MT_002,8)
(MT_003,0)