一.前言
为什么会有流计算?
因为有流^_^
为什么会有流?
这个原因比较复杂,部分是因为RDBMS解决不了,部分是为了解耦,部分是因为数据价值密度太低…,部分是因为架构^_^
二.概念
窗口:指定了时间长度和延时长度的时间段(长度也就是时间),比如:长度是5分钟,并且允许有1分钟的延迟。
窗口的创建时间:第一个元素到达就创建。
窗口的销毁时间:在窗口时间长度+延时长度到时。
每个窗口都会绑定一个触发器和一个执行函数,触发器定义了何时会触发窗口的执行函数的计算,比如水位经过了窗口结束时间的时候
窗口产生的目的就是为了计算一段短时间内的实时数据,你要计算一个月或一周的数据,不用用窗口,直接database批处理即可
用不用窗口,用啥窗口关键看需求:比如我要知道最近5分钟的最大人流量,那就适合用滑动窗口;我要知道用户在网站浏览行为,那就适合用会话窗口;如果需要知道每5分钟的最高气温/和,适合用滚动窗口。
下文描述的实验,使用kafka发送数据,每条消息数据格式 KEY:VAL;KEY2:VAL2;…KEYn:VALn;
然后使用spark的滑动窗口计算每一分钟(相当于滚动窗口)的每个KEY的值的和。
三.程序
SparkKafkaDemo.java
package org.ips.demo;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.*;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaInputDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka010.ConsumerStrategies;
import org.apache.spark.streaming.kafka010.KafkaUtils;
import org.apache.spark.streaming.kafka010.LocationStrategies;
import scala.Tuple2;
import java.util.*;
public class SparkKafkaDemo {
public static void main(String[] args) throws InterruptedException {
SparkConf conf = new SparkConf();
conf.setAppName("SparkKafkaDemo");
//conf.setMaster("local[4]");
//创建Spark流应用上下文,batch设为1min
JavaStreamingContext streamingContext = new JavaStreamingContext(conf,new Duration(1*60*1000));
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put("bootstrap.servers", "192.168.100.101:9092");
kafkaParams.put("key.deserializer", StringDeserializer.class);
kafkaParams.put("value.deserializer", StringDeserializer.class);
kafkaParams.put("group.id", "GROUP_SPARK_SUM");
kafkaParams.put("auto.offset.reset", "earliest");
Collection<String> topics = Arrays.asList("IPS_CASH");
final JavaInputDStream<ConsumerRecord<String, String>> stream =
KafkaUtils.createDirectStream(streamingContext,
LocationStrategies.PreferConsistent(),
ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams));
JavaDStream<String> wordsDS = stream.flatMap(new FlatMapFunction<ConsumerRecord<String,String>, String>() {
public Iterator<String> call(ConsumerRecord<String, String> r) throws Exception {
String value = r.value();
List<String> list = new ArrayList<String>();
String[] arr = value.split(";");
for (String s : arr) {
list.add(s);
}
return list.iterator();
}
});
//映射成元组,(KEY,近1min VAL)
JavaPairDStream<String,Integer> pairDS = wordsDS.mapToPair(new PairFunction<String, String, Integer>() {
public Tuple2<String, Integer> call(String s) throws Exception {
//KEY:VAL
String[] arr=s.split(":");
return new Tuple2<String,Integer>(arr[0],Integer.parseInt(arr[1]));
} }) ;
//聚合reduceByKeyAndWindow(KEY,近1minVAL),窗口长度设为1min,滑动间隔设为1min
JavaPairDStream<String,Integer> countDS = pairDS.reduceByKeyAndWindow(new Function2<Integer, Integer, Integer>() {
public Integer call(Integer v1, Integer v2) throws Exception {
return v1 + v2;
}
},new Duration(1*60*1000),new Duration(1*60*1000));
//打印
countDS.print();
streamingContext.start();
streamingContext.awaitTermination();
streamingContext.stop();
}}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.ips</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
<build>
<finalName>SparkKafkaDemo</finalName>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>org.ips.demo.SparkKafkaDemo</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
</project>
四.编译
mvn -U clean package assembly:assembly -Dmaven.test.skip=true
五.运行
5.1 提交任务
spark-submit --class org.ips.demo.SparkKafkaDemo --deploy-mode client --master yarn /home/test/SparkKafkaDemo.jar
…
5.2 使用kafka发消息
kafka-console-producer.sh --broker-list ipsnode1:9092 -topic IPS_CASH #相同颜色属于一分钟内发送
>a:60;b:20;c:30;
>a:61;b:45;c:10;
>a:100;b:200;c:300;
>a:111;b:222;c:333;
>a:1000;b:2000;c:3000;
>a:1001;b:1002;c:1003;
5.3 查看5.1的界面输出结果
…
2020-03-04 13:12:01 INFO DAGScheduler:54 - Job 3 finished: print at SparkKafkaDemo.java:63, took 0.091685 s
-------------------------------------------
Time: 1591247520000 ms
-------------------------------------------
(b,65)
(a,121)
(c,40)
…
2020-03-04 13:14:00 INFO DAGScheduler:54 - Job 7 finished: print at SparkKafkaDemo.java:63, took 0.066255 s
-------------------------------------------
Time: 1591247640000 ms
-------------------------------------------
(b,422)
(a,211)
(c,633)
…
2020-03-04 13:15:00 INFO DAGScheduler:54 - Job 9 finished: print at SparkKafkaDemo.java:63, took 0.100732 s
-------------------------------------------
Time: 1591247700000 ms
-------------------------------------------
(b,3002)
(a,2001)
(c,4003)
…