一、Spark Streaming介绍 ----------------------------------------------------------- 1.介绍 是spark core的扩展,针对实时数据的实时流处理技术 具有可扩展、高吞吐量、容错的特点 数据可以是来自于kafka,flume,tcpsocket,使用高阶函数(map reduce filter ,join , windows),处理的数据结果可以推送到database,hdfs 数据流处理还可以应用到机器学习和图计算中 2.内部执行流程: spark接受实时数据流,分成batch(分批次)进行处理,最终在每个batch终产生结果stream. 3.discretized stream or DStream, 离散流,表示的是连续的数据流。 DStream可通过kafka、flume等输入数据流产生,也可以通过对其他DStream进行高阶变换产生(类似于RDD)。 在内部,DStream是表现为RDD序列。 二、SparkStreaming的编程实现 ------------------------------------------------------------ 1.添加依赖 <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>2.1.0</version> </dependency> 2.编写scala -- SparkStreamingDemo
package com.test.spark.streaming.scala
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
object SparkStreamingDemo {
def main(args: Array[String]): Unit = {
//注意local[n] n > 1
val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
//创建sparkstreaming上下文一秒钟一个批次
val ssc = new StreamingContext(conf, Seconds(1))
//创建套接字文本流
val lines = ssc.socketTextStream("lcoalhost", 9999)
//
val words = lines.flatMap(_.split(" "))
//
val pairs = words.map(word => (word, 1))
//
val wordCounts = pairs.reduceByKey(_ + _)
//
wordCounts.print()
//启动
ssc.start() // Start the computation
//等待结束
ssc.awaitTermination() // Wait for the computation to terminate
}
}
3.先启动nc服务器,产生数据源 cmd> nc -l -p 9999 4.启动scala程序,查看结果 run 5.打包,丢到ubuntu上运行 $> nc -lk 9999 $> spark-submit --master spark://s100:7077 --name netwc --class com.test.spark.streaming.scala.SparkStreamingDemo TestSpark-1.0-SNAPSHOT.jar 6.java版
package com.test.spark.streaming.java;
import org.apache.spark.*;
import org.apache.spark.api.java.function.*;
import org.apache.spark.streaming.*;
import org.apache.spark.streaming.api.java.*;
import scala.Tuple2;
import scala.actors.threadpool.Arrays;
public class WCDemoJava {
public static void main(String [] args)
{
SparkConf conf = new SparkConf().setMaster("spark://s100:7077).setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));
JavaReceiverInputDStream<String> lines = jssc.socketTextStream("s100", 9999);
JavaDStream<String> words = lines.flatMap(x -> Arrays.asList(x.split(" ")).iterator());
JavaPairDStream<String, Integer> pairs = words.mapToPair(s -> new Tuple2<>(s, 1));
JavaPairDStream<String, Integer> wordCounts = pairs.reduceByKey((i1, i2) -> i1 + i2);
wordCounts.print();
jssc.start(); // Start the computation
try {
jssc.awaitTermination(); // Wait for the computation to terminate
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
三、DStream -- 离散流介绍 ------------------------------------------------------------ [注意事项] 1.启动上下文之后,不能启动新的流或者添加新的 2.上下文停止后不能restart. 3.同一JVM只有一个active的streamingcontext 4.停止streamingContext会一同stop掉SparkContext,如若只停止StreamingContext. ssc.stop(false|true); 5.SparkContext可以创建多个StreamingContext,创建新的之前要停掉旧的。 四、Receiver -- 接收者 ---------------------------------------------------------- 1.介绍 Receiver是接收者,从source接受数据,存储在内存中,供spark处理。 2.源 基本源:fileSystem | socket,内置API支持 高级源:kafka | flume | ...,需要引入pom.xml依赖 3.注意 使用local模式时,不能使用一个线程.使用的local[n],n需要大于receiver的个数。 五、SparkStreaming集成Kafka ------------------------------------------------------------- 1.启动kafka集群 a.启动zk b.启动kafka[s200,s300,s400] $> /soft/kafka/bin/kafka-server-start.sh -daemon /soft/kafka/config/server.properties c.验证kafka是否启动成功 $> netstat -ano | grep 9092 d.创建kafka主题 $> kafka-topics.sh --create --zookeeper s100:2181 --replication-factor 3 --partitions 3 --topic sparktopic1 2.引入pom.xml ... <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka-0-10_2.11</artifactId> <version>2.1.0</version> </dependency> 3.编写java代码
package com.test.spark.kafka;
import java.util.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.*;
import org.apache.spark.streaming.kafka010.*;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import scala.Tuple2;
public class SparkKafkaDemo {
public static void main(String [] args)
{
SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(10));
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put("bootstrap.servers", "s200:9092,s300:9092,s400:9092");
kafkaParams.put("key.deserializer", StringDeserializer.class);
kafkaParams.put("value.deserializer", StringDeserializer.class);
kafkaParams.put("group.id", "g6");
kafkaParams.put("auto.offset.reset", "latest");
kafkaParams.put("enable.auto.commit", false);
Collection<String> topics = Arrays.asList("sparktopic1");
JavaInputDStream<ConsumerRecord<String, String>> stream =
KafkaUtils.createDirectStream(
jssc,
LocationStrategies.PreferConsistent(),
ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)
);
JavaDStream js = stream.map(
new Function<ConsumerRecord<String, String>, String>() {
@Override
public String call(ConsumerRecord<String, String> v1) throws Exception {
return v1.value();
}
}
);
JavaDStream js1 = js.flatMap(new FlatMapFunction<String, String>() {
@Override
public Iterator<String> call(String s) throws Exception {
List<String> list = new ArrayList<String>();
String [] strs = s.split(" ");
for (String ss : strs){
list.add(ss);
}
return list.iterator();
}
});
JavaPairDStream<String,Integer> js2 = js1.mapToPair( x -> new Tuple2<>(x,1));
JavaPairDStream<String,Integer> js3 = js2.reduceByKey((x, y) -> x + y);
js3.print();
jssc.start();
try {
jssc.awaitTermination();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
4.开始kafka控制台生产者,测试 $> kafka-console-producer.sh --broker-list s200:9092 --topic sparktopic1 六、updateStateByKey函数 ----------------------------------------------------------------- 1.对DStream的每个key,value应用更新函数,从而产生新的DStream 2.分两步 a.定义状态 --- 状态可以是任意类型 b.定义状态更新函数 3.解释 比如单词统计,进行到maptopair将单词变成tiple<"word",1>,对变换的DStream应用 updateStateByKey函数 函数对应的法则是保持key<"word">原来的1的状态,一旦再出现key<'word'>的时候,就对状态值+1,kv就变成<'word',2>进行返回了 因为是流计算,是分批次进行计算的,所以,key的状态会跨批次的,一直应用法则进行更新和迭代 这样统计出来的单词就不是单批次的单词数量,而是从开始到现在,所有批次的总数量了 4.代码演示
package com.test.spark.streaming.java;
import org.apache.spark.*;
import org.apache.spark.api.java.Optional;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.streaming.*;
import org.apache.spark.streaming.api.java.*;
import scala.Tuple2;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class WCDemoJava {
public static void main(String [] args)
{
SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(10));
JavaReceiverInputDStream<String> lines = jssc.socketTextStream("localhost", 9999);
JavaDStream<String> words = lines.flatMap(
new FlatMapFunction<String, String>() {
@Override
public Iterator<String> call(String s) throws Exception {
List<String> list = new ArrayList<String>();
String [] strs = s.split(" ");
for (String ss : strs){
list.add(ss);
}
return list.iterator();
}
}
);
jssc.checkpoint("file:///d:\\share\\spark\\checkpoint");
JavaPairDStream<String, Integer> pairs = words.mapToPair(s -> new Tuple2<>(s, 1));
JavaPairDStream<String, Integer> pairs1 = pairs.updateStateByKey(new Function2<List<Integer>, Optional<Integer>, Optional<Integer>>() {
public Optional<Integer> call(List<Integer> v1, Optional<Integer> v2) throws Exception {
//取得以前的状态
Integer newState = v2.isPresent()? v2.get() : 0;
System.out.println("oldState : " + newState );
for (Integer i : v1) {
//更新旧状态[加上这个批次出现的次数]
newState = newState + i;
}
return Optional.of(newState);
}
});
pairs1.print();
jssc.start(); // Start the computation
try {
jssc.awaitTermination(); // Wait for the computation to terminate
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
七、Streaming Windows 窗口化操作,跨批次 ----------------------------------------------------- 1.batch interval : 批次间隔,最小的时间单位 2.window length : 窗口长度,跨批次,批次的正数倍长度 3.silder interval: 滑块间隔,窗口计算的间隔时间,多长时间计算一次窗口,批次的整倍数。也就是相邻窗口之间的间隔[多少个批次] 4.常用窗口化操作: a.window(windowLength, slideInterval) //Return a new DStream which is computed based on windowed batches of the source DStream. //应用于非pair的DS,返回一个新的包含窗口内所有批次的DS b.countByWindow(windowLength, slideInterval) //Return a sliding window count of elements in the stream. //应用于非pair的DS,统计窗口内所有批次的元素的个数 c.reduceByWindow(func, windowLength, slideInterval) //Return a new single-element stream, created by aggregating elements in the stream over a sliding interval using func. //The function should be associative and commutative so that it can be computed correctly in parallel. //应用于非pair的DS, 将窗口内所有批次的元素,按照func进行聚合 d.reduceByKeyAndWindow(func, windowLength, slideInterval, [numTasks]) //When called on a DStream of (K, V) pairs, returns a new DStream of (K, V) pairs //where the values for each key are aggregated using the given reduce function func over batches in a sliding window. //Note: By default, this uses Spark's default number of parallel tasks (2 for local mode, //and in cluster mode the number is determined by the config property spark.default.parallelism) to do the grouping. //You can pass an optional numTasks argument to set a different number of tasks. //应用于pair<k,v>的DS,将窗口内,key相同的value按照func进行聚合操作 e.reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, [numTasks]) //A more efficient version of the above reduceByKeyAndWindow() //where the reduce value of each window is calculated incrementally using the reduce values of the previous window. //This is done by reducing the new data that enters the sliding window, //and “inverse reducing” the old data that leaves the window. //An example would be that of “adding” and “subtracting” counts of keys as the window slides. //However, it is applicable only to “invertible reduce functions”, //that is, those reduce functions which have a corresponding “inverse reduce” function (taken as parameter invFunc). //Like in reduceByKeyAndWindow, the number of reduce tasks is configurable through an optional argument. //Note that checkpointing must be enabled for using this operation. f.countByValueAndWindow(windowLength, slideInterval, [numTasks]) //When called on a DStream of (K, V) pairs, //returns a new DStream of (K, Long) pairs where the value of each key is its frequency within a sliding window. //Like in reduceByKeyAndWindow, the number of reduce tasks is configurable through an optional argument. //应用于pair<k,v>的DS,将窗口内,所有批次的pair<k,v>作为key,统计出现的次数作为value,进行返回 5.示例 a.reduceByKeyAndWindow(): // Reduce last 30 seconds of data, every 10 seconds //windows length = 30 //slider interval = 10 scala> val windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) => (a + b), Seconds(30), Seconds(10)) // Reduce last 30 seconds of data, every 10 seconds java> JavaPairDStream<String, Integer> windowedWordCounts = pairs.reduceByKeyAndWindow((i1, i2) -> i1 + i2, Durations.seconds(30), Durations.seconds(10)); b.代码
package com.test.spark.streaming.java;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.Optional;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.Seconds;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import scala.Some;
import scala.Tuple2;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class WCSparkDemo1 {
public static void main(String[] args) throws Exception {
SparkConf conf = new SparkConf();
conf.setAppName("wc");
conf.setMaster("local[4]");
//创建Spark流应用上下文
JavaStreamingContext jsc = new JavaStreamingContext(conf, Seconds.apply(5));
jsc.checkpoint("file:///d:/share/spark/check");
//创建socket离散流
JavaReceiverInputDStream sock = jsc.socketTextStream("localhost",9999);
//压扁
JavaDStream<String> wordsDS = sock.flatMap(new FlatMapFunction<String,String>() {
public Iterator call(String str) throws Exception {
List<String> list = new ArrayList<String>() ;
String[] arr = str.split(" ");
for(String s : arr){
list.add(s);
}
return list.iterator();
}
});
JavaDStream<String> wordsDS1 = wordsDS.window(Seconds.apply(15), Seconds.apply(10));
wordsDS1.print();
// //窗口内的单词总个数统计[1个批次是5秒,3个批次为一个窗口,2个批次的窗口间隔 : 10秒计算一次,每次计算15秒内的批次值]
// JavaDStream js1 = wordsDS.countByWindow(Seconds.apply(15), Seconds.apply(10));
// js1.print();
//映射成元组
// JavaPairDStream<String,Integer> pairDS = wordsDS.mapToPair(new PairFunction<String, String, Integer>() {
// public Tuple2<String, Integer> call(String s) throws Exception {
// return new Tuple2<String,Integer>(s,1);
// }
// }) ;
//窗口统计,将传入的pair元素对作为元素,value为统计的元素对的个数
//输入
//hello wordl
//hello wordl
//hello wordl
//hello wordl
//hello wordl
//输出
// ((hello,1),5)
// ((wordl,1),5)
// JavaPairDStream countDS = pairDS.countByValueAndWindow(Seconds.apply(15), Seconds.apply(10));
// countDS.print();
// //key窗口聚合[1个批次是5秒,3个批次为一个窗口,2个批次的窗口间隔 : 10秒计算一次,每次计算15秒内的批次值]
// JavaPairDStream<String,Integer> countDS = pairDS.reduceByKeyAndWindow(new Function2<Integer, Integer, Integer>() {
// public Integer call(Integer v1, Integer v2) throws Exception {
// return v1 + v2;
// }
// },Seconds.apply(15), Seconds.apply(10));
//打印
// countDS.print();
jsc.start();
jsc.awaitTermination();
jsc.stop();
}
}
八、生产环境下,SparkStream容错的实现 ------------------------------------------------------------- 1.两个概念 a.Spark Driver //驱动,运行用户编写的程序代码的主机。 b.Executors //执行的spark driver提交的job,内部含有附加组件比如receiver //receiver接受数据并以block方式保存在memory中,同时,将数据块复制到其他Executor中,以备于容错 //每个批次末端会形成新的DStream,交给下游处理 //如果receiver故障,其他执行器中的receiver会启动进行数据的接收 2.故障类型以及解决办法 a.如果Executor故障,所有未被处理的数据都会丢失,解决办法可以通过wal(写前日志:hbase,hdfs/WALs都有使用)方式将数据预先写入到hdfs或者s3中进行储存. b.如果Driver故障,driver程序就会停止,所有executor都是丢失连接,停止计算过程。解决办法需要配置和编程 3.解决Driver故障 a.第一步,配置Driver程序自动重启,使用特定的集群管理器clustermanager实现。 b.第二步,重启时,从宕机的地方进行重启,通过检查点机制可以实现该功能。 1)创建检查点 //目录可以是本地,可以是hdfs. jsc.checkpoint("d://...."); 2)使用JavaStreamingContext.getOrCreate方式创建上下文 //不再使用new方式创建SparkStreamContext对象, //而是通过工厂方式JavaStreamingContext.getOrCreate()方法创建上下文对象, //首先会检查检查点目录,看是否有job运行,没有就new新的。 JavaStreamingContext jsc = JavaStreamingContext.getOrCreate("file:///d:/scala/check", new Function0<JavaStreamingContext>() { public JavaStreamingContext call() throws Exception { JavaStreamingContext jsc = new JavaStreamingContext(conf, Seconds.apply(2)); jsc.checkpoint("file:///d:/scala/check"); return jsc; } }); c.编写容错测试代码,计算过程要编写到Function0的call方法中。
package com.test.spark.streaming.java;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function0;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
public class SparkStreamingFaultTolerant {
public static void main(String [] args)
{
//创建一个JavaStreamingContext工厂
Function0<JavaStreamingContext> contextFactory = new Function0<JavaStreamingContext>() {
//首次创建context时调用该方法
public JavaStreamingContext call() {
SparkConf conf = new SparkConf();
conf.setMaster("local[4]");
conf.setAppName("wc");
JavaStreamingContext jssc = new JavaStreamingContext(conf,new Duration(2000));
JavaDStream<String> lines = jssc.socketTextStream("localhost",9999);
/******* 变换代码放到此处 ***********/
JavaDStream<Long> dsCount = lines.countByWindow(new Duration(24 * 60 * 60 * 1000),new Duration(2000));
dsCount.print();
//设置检察点目录
jssc.checkpoint("file:///d:/share/spark/checkpoint");
return jssc;
}
};
//失败重建时会经过检查点。
JavaStreamingContext context = JavaStreamingContext.getOrCreate("file:///d:/share/spark/checkpoint", contextFactory);
context.start();
try {
context.awaitTermination();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}