一、reduceByKey和groupByKey的区别
1、reduceByKey:按照 key进行聚合,在 shuffle 之前有 combine(预聚合)操作,返回结果是 RDD[k,v]。
2、groupByKey:按照 key进行分组,直接进行 shuffle。开发指导:reduceByKey比 groupByKey,建议使用。但是需要注意是否会影响业务逻辑。
1、reduceByKey(func):使用 func 函数合并具有相同键的值。
val list = List("hadoop","spark","hive","spark")
val rdd=sc.parallelize(list)
val pairRdd= rdd.map((_,1))
pairRdd.reduceByKey(_+_).collect.foreach(println)
上例中,我们先是建立了一个 list,然后建立通过这个 list 集合建立一个 rdd;然后我们通过 map 函数将 list 的 rdd 转化成键值对形式的 rdd;然后我们通过 reduceByKey 方法对具有相同 key 的值进行 func(_+_)的累加操作。
(hive,1)
(spark,2)
(hadoop,1)
list: List[String]=List(hadoop, spark, hive, spark)
rdd: org.apache.spark.rdd.RDD[String]= ParallelCollectionRDD[127] at parallelize at command-3434610298353610:2pairRdd: org.apache.spark.rdd.RDD[(String, Int)]= MapPartitionsRDD[128] at map at command-3434610298353610:3
pairRdd.collect.foreach(println)//打印pairRdd
(hive,1)
(spark,1)
(hadoop,1)
(spark,1)
我们需要留意的事情是,我们调用了reduceByKey操作的返回的结果类型是
org.apache.spark.rdd.RDD[(String, Int)]
注意,我们这里的collect()方法的作用是收集分布在各个worker的数据到driver节点。
如果不使用这个方法,每个worker的数据只在自己本地显示,并不会在driver节点显示。
2、groupByKey():对具有相同key的value进行分组。
val list = List("hadoop","spark","hive","spark")
val rdd=sc.parallelize(list)
val pairRdd= rdd.map(x => (x,1))
pairRdd.groupByKey().collect.foreach(println)
得出的结果为
(hive,CompactBuffer(1))
(spark,CompactBuffer(1, 1))
(hadoop,CompactBuffer(1))
list: List[String]=List(hadoop, spark, hive, spark)
rdd: org.apache.spark.rdd.RDD[String]= ParallelCollectionRDD[130] at parallelize at command-3434610298353610:2pairRdd: org.apache.spark.rdd.RDD[(String, Int)]= MapPartitionsRDD[131] at map at command-3434610298353610:3
CompactBuffer:CompactBuffer并不是scala里定义的数据结构,而是spark里的数据结构,它继承自一个迭代器和序列,所以它的返回值是一个很容易进行循环遍历的集合。
可以看到,结果并不是把具有相同key值进行相加,而是就简单的进行了分组,生成一个sequence。因此,我们可以把groupByKey()当作reduceByKey(func)操作的一部分,reduceByKey(func)先是对rdd进行groupByKey()然后在对每个分组进行func操作。
pairRdd.reduceByKey(_+_).collect.foreach(println)
这里通过groupByKey()后调用map遍历每个分组,然后通过t => (t._1,t._2.sum)对每个分组的值进行累加。因为groupByKey()操作是把具有相同类型的key收集到一起聚合成一个集合,集合中有个sum方法,对所有元素进行求和。
注意:(k,v)形式的数据,我们可以通过 ._1,._2 来访问键和值,用占位符表示就是 _._1,_._2,这里前面的两个下划线的含义是不同的,前边下划线是占位符,后边的是访问方式。 我们记不记得 ._1,._2,._3 是元组的访问方式。我们可以把键值看成二维的元组。
3、区别:
reduceByKey()对于每个key对应的多个value进行了merge操作,最重要的是它能够先在本地进行merge操作。merge可以通过func自定义。
groupByKey()也是对每个key对应的多个value进行操作,但是只是汇总生成一个sequence,本身不能自定义函数,只能通过额外通过map(func)来实现。
使用reduceByKey()的时候,本地的数据先进行merge然后再传输到不同节点再进行merge,最终得到最终结果。
而使用groupByKey()的时候,并不进行本地的merge,全部数据传出,得到全部数据后才会进行聚合成一个sequence,
groupByKey()传输速度明显慢于reduceByKey()。
虽然groupByKey().map(func)也能实现reduceByKey(func)功能,但是,优先使用reduceByKey(func)
二、map和flatMap的区别:
三、repartition和partitionBy的区别:
repartition 和 partitionBy 都是对数据进行重新分区,默认都是使用 HashPartitioner,区别在于partitionBy 只能用于 PairRDD,但是当它们同时都用于 PairRDD时,结果却不一样:
不难发现,其实 partitionBy 的结果才是我们所预期的,我们打开 repartition 的源码进行查看:
/*** Return a new RDD that has exactly numPartitions partitions.
*
* Can increase or decrease the level of parallelism in this RDD. Internally, this uses
* a shuffle to redistribute data.
*
* If you are decreasing the number of partitions in this RDD, consider using `coalesce`,
* which can avoid performing a shuffle.
*
* TODO Fix the Shuffle+Repartition data loss issue described in SPARK-23207.*/def repartition(numPartitions: Int)(implicit ord: Ordering[T]= null): RDD[T] =withScope {undefined
coalesce(numPartitions, shuffle= true)
}/*** Return a new RDD that is reduced into `numPartitions` partitions.
*
* This results in a narrow dependency, e.g. if you go from 1000 partitions
* to 100 partitions, there will not be a shuffle, instead each of the 100
* new partitions will claim 10 of the current partitions. If a larger number
* of partitions is requested, it will stay at the current number of partitions.
*
* However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
* this may result in your computation taking place on fewer nodes than
* you like (e.g. one node in the case of numPartitions = 1). To avoid this,
* you can pass shuffle = true. This will add a shuffle step, but means the
* current upstream partitions will be executed in parallel (per whatever
* the current partitioning is).
*
* @note With shuffle = true, you can actually coalesce to a larger number
* of partitions. This is useful if you have a small number of partitions,
* say 100, potentially with a few partitions being abnormally large. Calling
* coalesce(1000, shuffle = true) will result in 1000 partitions with the
* data distributed using a hash partitioner. The optional partition coalescer
* passed in must be serializable.*/def coalesce(numPartitions: Int, shuffle: Boolean= false,
partitionCoalescer: Option[PartitionCoalescer]=Option.empty)
(implicit ord: Ordering[T]= null)
: RDD[T]=withScope {undefined
require(numPartitions> 0, s"Number of partitions ($numPartitions) must be positive.")if(shuffle) {/**Distributes elements evenly across output partitions, starting from a random partition.*/val distributePartition= (index: Int, items: Iterator[T]) =>{undefined
var position= newRandom(hashing.byteswap32(index)).nextInt(numPartitions)
items.map { t=>
//Note that the hash code of the key will just be the key itself. The HashPartitioner//will mod it with the number of total partitions.
position = position + 1(position, t)
}
} : Iterator[(Int, T)]//include a shuffle step so that our upstream tasks are still distributed
newCoalescedRDD(newShuffledRDD[Int, T, T](mapPartitionsWithIndex(distributePartition),newHashPartitioner(numPartitions)),
numPartitions,
partitionCoalescer).values
}else{new CoalescedRDD(this, numPartitions, partitionCoalescer)
}
}
即使是RairRDD也不会使用自己的key,repartition 其实使用了一个随机生成的数来当做 Key,而不是使用原来的 Key!!
————————————————
原文转自:spark算子之间的区别_spark常见算子的区别_黑岛人的博客-CSDN博客https://blog.csdn.net/weixin_29531989/article/details/111895318?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522164716453016780274192152%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=164716453016780274192152&b