#spark学习笔记
##tips
spark sql
spark-sql --executor-memory 4G --total-executor-cores 20
如果在eclipse上直接调试
conf.setMaster("local");可以直接在本地调试 ;也可设置成本地模式
##调试WordCount
先看一下WordCount的代码
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
object WordCount {
def main(args: Array[String]) {
val conf = new SparkConf();
conf.setAppName("WordCount").setMaster("spark://hadoopallinone:7077");
val sc = new SparkContext(conf) ;
val file = "hdfs://hadoopallinone:9000/user.txt" ;
val lines = sc.textFile(file);
val words = lines.flatMap(_.split(" "));
val wordCount = words.countByValue();
println(wordCount);
}
}
###spark-submit
spark-submit --class WordCount --master spark://hadoopallinone:7077 ~/jars/WordCount.jar
##spark运行架构
###几个基本概念:
(1)job:包含多个task组成的并行计算,往往由action催生。
(2)stage:job的调度单位。
(3)task:被送到某个executor上的工作单元。
(4)taskSet:一组关联的,相互之间没有shuffle依赖关系的任务组成的任务集。
一个应用程序由一个driver program和多个job构成。一个job由多个stage组成。一个stage由多个没有shuffle关系的task组成。
###spark应用程序的运行架构
(1)简单的说:
由driver向集群申请资源,集群分配资源,启动executor。driver将spark应用程序的代码和文件传送给executor。executor上运行task,运行完之后将结果返回给driver或者写入外界。
(2)复杂点说:
提交应用程序,构建sparkContext,构建DAG图,提交给scheduler进行解析,解析成一个个stage,提交给集群,由集群任务管理器进行调度,集群启动spark executor。driver把代码和文件传给executor。executor进行各种运算完成task任务。driver上的block tracker记录executor在各个节点上产生的数据块。task运行完之后,将数据写入HDFS上或者其他类型数据库里。
(3)全面点说:
spark应用程序进行各种transformation的计算,最后通过action触发job。提交之后首先通过sparkContext根据RDD的依赖关系构建DAG图,DAG图提交给DAGScheduler进行解析,解析时是以shuffle为边界,反向解析,构建stage,stage之间也有依赖关系。这个过程就是对DAG图进行解析划分stage,并且计算出各个stage之间的依赖关系。然后将一个个TaskSet提交给底层调度器,在spark中是提交给taskScheduler处理,生成TaskSet manager,最后提交给executor进行计算,executor多线程计算,计算完反馈给TaskSetmanager,再反馈给taskScheduler,然后再反馈回DAGScheduler。全部运行完之后写入数据。
(4)更加深入理解:
应用程序提交后,触发action,构建sparkContext,构建DAG图,提交给DAGScheduler,构建stage,以stageSet方式提交给TaskScheduler,构建taskSet Manager,然后将task提交给executor运行。executor运行完task后,将完成信息提交给schedulerBackend,由它将任务完成的信息提交给TaskScheduler。TaskScheduler反馈信息给TaskSetManager,删除该task任务,执行下一个任务。同时TaskScheduler将完成的结果插入到成功队列里,加入之后返回加入成功的信息。TaskScheduler将任务处理成功的信息传给TaskSet Manager。全部任务完成后TaskSetManager将结果反馈给DAGScheduler。如果属于resultTask,交给JobListener。如果不属于resultTask,保存结果。
##sparksql汇总
###sparksql和hive结合
spark增加hive的配置
cd $SPARK_HOME/conf/
vim hive-site.xml
hive.metastore.uris
thrift://hadoopallinone:9083
Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.
启动hive
hive --service metastore &
启动spark
$SPARK_HOME/sbin/start-all.sh
spark sql
$SPARK_HOME/bin/spark-sql --master spark://hadoopallinone:7077 --name testsparksql --executor-cores 1 --executor-memory 512m
show databases;
###sparksql代码开发
和hive结合
import org.apache.spark.sql.SQLContext
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
val sparkConf = new SparkConf()
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val result = sqlContext.sql("show databases").collect.foreach(println)
内存创建RDD
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.functions._
// One method for defining the schema of an RDD is to make a case class with the desired column
// names and types.
case class Record(key: Int, value: String)
object RDDRelation {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("RDDRelation").setMaster("local");
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
// Importing the SQL context gives access to all the SQL functions and implicit conversions.
import sqlContext.implicits._
val df = sc.parallelize((1 to 100).map(i => Record(i, s"val_$i"))).toDF()
// Any RDD containing case classes can be registered as a table. The schema of the table is
// automatically inferred using scala reflection.
df.registerTempTable("records")
// Once tables have been registered, you can run SQL queries over them.
println("Result of SELECT *:")
sqlContext.sql("SELECT * FROM records").collect().foreach(println)
// Aggregation queries are also supported.
val count = sqlContext.sql("SELECT COUNT(*) FROM records").collect().head.getLong(0)
println(s"COUNT(*): $count")
// The results of SQL queries are themselves RDDs and support all normal RDD functions. The
// items in the RDD are of type Row, which allows you to access each column by ordinal.
val rddFromSql = sqlContext.sql("SELECT key, value FROM records WHERE key s"Key: ${row(0)}, Value: ${row(1)}").collect().foreach(println)
// Queries can also be written using a LINQ-like Scala DSL.
df.where($"key" === 1).orderBy($"value".asc).select($"key").collect().foreach(println)
// Write out an RDD as a parquet file.
df.write.parquet("pair.parquet")
// Read in parquet file. Parquet files are self-describing so the schmema is preserved.
val parquetFile = sqlContext.read.parquet("pair.parquet")
// Queries can be run using the DSL on parequet files just like the original RDD.
parquetFile.where($"key" === 1).select($"value".as("a")).collect().foreach(println)
// These files can also be registered as tables.
parquetFile.registerTempTable("parquetFile")
sqlContext.sql("SELECT * FROM parquetFile").collect().foreach(println)
sc.stop()
}
}
// scalastyle:on println
##hive on spark
$SPARK_HOME/sbin/start-thriftserver.sh --master spark://hadoopallinone:7077 --name hive --executor-cores 1 --executor-memory 512m
$SPARK_HOME/bin/beeline -n hadoop -u jdbc:hive2://hadoopallinone:10000/
此时 spark thriftserver 提供了一个可以供jdbc链接的服务,可以使用java scala进行编程
set hive.execution.engine=spark;
##spark RDD
1,transformation是得到一个新的RDD,方式很多,比如从数据源生成一个新的RDD,从RDD生成一个新的RDD
2,action是得到一个值,或者一个结果(直接将RDDcache到内存中)所有的transformation都是采用的懒策略,就是如果只是将transformation提交是不会执行计算的,计算只有在action被提交的时候才被触发。
###transformation操作
map(func):对调用map的RDD数据集中的每个element都使用func,然后返回一个新的RDD,这个返回的数据集是分布式的数据集
filter(func): 对调用filter的RDD数据集中的每个元素都使用func,然后返回一个包含使func为true的元素构成的RDD
flatMap(func):和map差不多,但是flatMap生成的是多个结果
mapPartitions(func):和map很像,但是map是每个element,而mapPartitions是每个partition
mapPartitionsWithSplit(func):和mapPartitions很像,但是func作用的是其中一个split上,所以func中应该有index
sample(withReplacement,faction,seed):抽样
union(otherDataset):返回一个新的dataset,包含源dataset和给定dataset的元素的集合
distinct([numTasks]):返回一个新的dataset,这个dataset含有的是源dataset中的distinct的element
groupByKey(numTasks):返回(K,Seq[V]),也就是hadoop中reduce函数接受的key-valuelist
reduceByKey(func,[numTasks]):就是用一个给定的reducefunc再作用在groupByKey产生的(K,Seq[V]),比如求和,求平均数
sortByKey([ascending],[numTasks]):按照key来进行排序,是升序还是降序,ascending是boolean类型
join(otherDataset,[numTasks]):当有两个KV的dataset(K,V)和(K,W),返回的是(K,(V,W))的dataset,numTasks为并发的任务数
cogroup(otherDataset,[numTasks]):当有两个KV的dataset(K,V)和(K,W),返回的是(K,Seq[V],Seq[W])的dataset,numTasks为并发的任务数
cartesian(otherDataset):笛卡尔积就是m*n,大家懂的
action操作
reduce(func):说白了就是聚集,但是传入的函数是两个参数输入返回一个值,这个函数必须是满足交换律和结合律的
collect():一般在filter或者足够小的结果的时候,再用collect封装返回一个数组
count():返回的是dataset中的element的个数
first():返回的是dataset中的第一个元素
take(n):返回前n个elements,这个士driverprogram返回的
takeSample(withReplacement,num,seed):抽样返回一个dataset中的num个元素,随机种子seed
saveAsTextFile(path):把dataset写到一个textfile中,或者hdfs,或者hdfs支持的文件系统中,spark把每条记录都转换为一行记录,然后写到file中
saveAsSequenceFile(path):只能用在key-value对上,然后生成SequenceFile写到本地或者hadoop文件系统
countByKey():返回的是key对应的个数的一个map,作用于一个RDD
foreach(func):对dataset中的每个元素都使用func
##spark和hbase整合
将$HBASE_HOME/conf/hbase-site.xml 链接到 $SPARK_HOME/conf/
ln -s $HBASE_HOME/conf/hbase-site.xml $SPARK_HOME/conf/
spark-shell --master spark://hadoopallinone:7077 --name testhbase --executor-cores 1 --executor-memory 512m
##spark streaming
###监控文件夹
import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
object StreamingWordCount {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("StreamingWordCount").setMaster("local")
val ssc = new StreamingContext(conf,Seconds(20))
val lines = ssc.textFileStream("d:/test/")
val counts = lines.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_)
counts.print()
val map = lines.flatMap(_.split(" ")).map(x=>(x,1))
map.print()
ssc.start()
ssc.awaitTermination()
}
}
###建立一个socket server 推送数据
import java.util.Random
import scala.io.Source
import java.net.ServerSocket
import java.io.PrintWriter
object SaleSimulation {
val rdm = new Random
def index(length:Int)={
rdm.nextInt(length)
}
def main(args: Array[String]): Unit = {
if (args.length!=3) {
println("Useage: ")
}
val filename = args(0)
var lines = Source.fromFile(filename).getLines().toList
val filerow = lines.length
val listener = new ServerSocket(args(1).toInt)
while(true){
val socket = listener.accept()
new Thread(){
override def run(){
println("发送消息到"+socket.getInetAddress)
val print = new PrintWriter(socket.getOutputStream,true)
while(true){
Thread.sleep(args(2).toLong)
val content = lines(index(filerow))
println(content)
print.write(content+"\n")
print.flush()
}
print.close()
socket.close()
}
}.run()
}
}
}
###spark streaming 接收网络数据
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.api.java.StorageLevels
object NetworkWordCount {
def main(args: Array[String]): Unit = {
if (args.length < 2) {
System.err.println("Usage: NetworkWordCount ")
System.exit(1)
}
val sparkConf = new SparkConf().setAppName("NetworkWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf,Seconds(5))
val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
val counts = lines.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_)
counts.print()
ssc.start()
ssc.awaitTermination()
}
}
###spark streaming 设置检查点
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.api.java.StorageLevels
object StateWordCount {
def main(args: Array[String]): Unit = {
if (args.length < 2) {
System.err.println("Usage: StateWordCount ")
System.exit(1)
}
def updateFunc = (values:Seq[Int],state:Option[Int]) =>{
val curcount = values.foldLeft(0)(_+_)
val precount = state.getOrElse(0)
Some(curcount+precount)
}
val sparkConf = new SparkConf().setAppName("StateWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf,Seconds(5))
ssc.checkpoint("hdfs://cloudera1:8020/checkpoint")
val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
val counts = lines.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_)
counts.updateStateByKey(updateFunc)
counts.print()
ssc.start()
ssc.awaitTermination()
}
}
###spark streaming window
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.api.java.StorageLevels
object WindowWordCount {
def main(args: Array[String]): Unit = {
if (args.length < 2) {
System.err.println("Usage: WindowWordCount ")
System.exit(1)
}
// 创建SparkConf实例
val sparkConf = new SparkConf().setAppName("WindowWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf,Seconds(5))
val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
val counts = lines.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKeyAndWindow((a:Int,b:Int)=>(a + b), Seconds(args(2).toInt), Seconds(args(3).toInt))
counts.print()
ssc.start()
ssc.awaitTermination()
}
}
###金额计算
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.api.java.StorageLevels
object SaleAmount {
def main(args: Array[String]): Unit = {
if (args.length < 2) {
System.err.println("Usage: SaleAmount ")
System.exit(1)
}
// 创建SparkConf实例
val sparkConf = new SparkConf().setAppName("SaleAmount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf,Seconds(5))
val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
val counts = lines.map(_.split(" ")).filter(_.length==2).map(x=>(1,x(1).toDouble)).reduceByKey(_+_)
counts.print()
ssc.start()
ssc.awaitTermination()
}
}