一 运维hadoop spark http://192.168.229.133:50070/dfshealth.html#tab-overview http://192.168.229.133:8080 停掉hadoop cd /usr/local/hadoop-2.6.5/sbin 停掉dfs[root@testceph sbin]# stop-dfs.sh 启动dfs[root@testceph sbin]# start-dfs.sh 查看日志文件 Starting namenodes on [testceph] testceph: starting namenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-testceph.out redis1: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-redis1.out testceph: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-testceph.out redis2: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-redis2.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-testceph.out [root@testceph sbin]# jps 7266 Master 7365 Worker 10150 SecondaryNameNode 10279 Jps 9836 NameNode 9983 DataNode [root@testceph sbin]# 启动spark,停止spark [root@testceph logs]# cd /usr/local/spark-2.2.0-bin-hadoop2.6/sbin/ [root@testceph sbin]# ./stop-all.sh [root@testceph sbin]# ./start-all.sh [root@testceph sbin]# cd ../bin/ #启动Spark shell命令行 [root@testceph bin]# ./spark-shell Spark context Web UI available at http://192.168.229.133:4040 scala> 二spark RDD 函数讲解与实战分析 什么是RDD RDD是Spark中的抽象数据结构类型,任何数据在Spark中都被表示为RDD。从编程的角度来看,RDD可以简单看成是一个数组。和普通数组的区别是,RDD中的数据是分区存储的,这样不同分区的数据就可以分布在不同的机器上,同时可以被并行处理。因此,Spark应用程序所做的无非是把需要处理的数据转换为RDD,然后对RDD进行一系列的变换和操作从而得到结果。 2.1 map(function) map是对RDD中的每个元素都执行一个指定的函数来产生一个新的RDD.任何原RDD中的元素在新RDD中都有且只有一个元素与之对应。 scala> val a = sc.parallelize(1 to 9, 3) scala> val b = a.map(x => x*2) # x=>x*2是一个函数,x是传入参数即RDD的每个元素,x*2是返回值 scala> a.collect res0: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9) scala> b.collect res1: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18) scala> 2.2map 也可以把key变成key-value对 scala> val a =sc.parallelize(List("dog", "tiger", "lion", "cat", "panther", "eagle"), 2) scala> val b = a.map(x => (x,1)) scala> b.collect.foreach(println(_)) (dog,1) (tiger,1) (lion,1) (cat,1) (panther,1) (eagle,1) scala> 2.3 mapPartitions(function) map()的输入函数是应用于RDD中每个元素, 而mapPartitions()的输入函数是应用于每个分区 2.4 mapValues(function) 原RDD中的Key保持不便,与新的Value一起组成新的RDD中的元素。因此,该函数只适用于元素未KV对的RDD scala> val a = sc.parallelize(List("dog", "tiger", "lion", "cat", "panther", " eagle"), 2) a: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[4] at parallelize at <console>:24 scala> val b = a.map(x => (x.length, x)) b: org.apache.spark.rdd.RDD[(Int, String)] = MapPartitionsRDD[5] at map at <console>:26 scala> b.mapValues("x" + _ + "x").collect res3: Array[(Int, String)] = Array((3,xdogx), (5,xtigerx), (4,xlionx), (3,xcatx), (7,xpantherx), (6,x eaglex)) scala> 2.5 mapWith mapWith是map的另外一个变种,map只需要一个输入函数,而mapWith有两个输入函数。它的定义如下 def mapWith[A: ClassTag, U: ](constructA: Int => A, preservesPartitioning: Boolean = false)(f: (T, A) => U): RDD[U] 第一个函数 constructA是把RDD的partition index(index从0开始)作为输入,输出为新类型A; 第二个函数f是把二元组(T,A)作为输入(其中T为原RDD中的元素,A为第一个函数的输出),输出类型为U scala> val x = sc.parallelize(List(1,2,3,4,5,6,7,8,9,10), 3) scala> x.mapWith(a => a * 10)((a, b) => (b + 2)).collect res4: Array[Int] = Array(2, 2, 2, 12, 12, 12, 22, 22, 22, 22) 2.6flatMap 函数 flatMapWith与mapWith很类似,都是接收两个函数,一个函数把partitionIndex作为输入,输出是一个新类型A:另外一个函数是以二元组(T,A)作为输入,输出为一个序列, 这样序列里面的元素组成了新的RDD scala> val a = sc.parallelize(1 to 5, 2) scala> val b = a.flatMap(x => 1 to x) scala> b.collect res6: Array[Int] = Array(1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5) scala> 2.7 flatMapValues函数 flatMapValues类似于mapValues,不同的在于flatMapValues应用于元素为KV对的RDD中Value.每个一元素的Value被输入函数映射为一系列的值, 然后这些值再与原RDD中的Key组成一系列新的KV对 scala> val a = sc.parallelize(List((1,2),(3,4),(3,6))) scala> val b = a.flatMapValues(x=>x.to(5)) scala> b.collect res7: Array[(Int, Int)] = Array((1,2), (1,3), (1,4), (1,5), (3,4), (3,5)) scala> 2.8 reduce 函数 reduce将RDD中元素两两传递给输入函数,同时产生一个新的值,新产生的值与RDD中下一个元素再被传递给输入函数直到最后只有一个值为止。 scala> val c = sc.parallelize(1 to 10) scala> c.reduce((x,y)=>x+y) res8: Int = 55 scala> 2.9 reduceByKey函数 reduceByKey就是对元素KV对的RDD中Key相同的元素的Value进行reduce,因此,key相同的多个元素的值被reduce为一个值,然后将原RDD中的Key组成一个新的KV对 scala> val a =sc.parallelize(List((1,2),(3,4),(3,6))) a: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[13] at parallelize at <console>:24 scala> a.reduceByKey((x,y) => x + y).collect res9: Array[(Int, Int)] = Array((1,2), (3,10)) 2.10 cartesian函数 scala> val x = sc.parallelize(List(1,2,3,4,5)) scala> val y = sc.parallelize(List(6,7,8,9,10)) scala> x.cartesian(y).collect res10: Array[(Int, Int)] = Array((1,6), (1,7), (2,6), (2,7), (1,8), (1,9), (1,10), (2,8), (2,9), (2,10), (3,6), (3,7), (4,6), (4,7), (5,6), (5,7), (3,8), (3,9), (3,10), (4,8), (4,9), (4,10), (5,8), (5,9), (5,10)) scala> 2.11 aggregate函数 scala> val z = sc.parallelize(List(1,2,3,4,5,6),2) z: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[19] at parallelize at <console>:24 scala> z.aggregate(0)(math.max(_,_),_+_) res11: Int = 9 scala> scala> val z = sc.parallelize(List("12", "23", "345", ""),2) scala> z.aggregate("")((x,y) => math.min(x.length, y.length).toString,(x,y) => x+y) res12: String = 10
三 Spark RDD函数讲解
最新推荐文章于 2023-08-17 17:47:35 发布