Spark
hellboy0621
怕什么真理无穷,进一寸有一寸的欢喜。
展开
-
【Spark学习笔记】1、Java-Maven-Spark项目环境搭建
现在学习个新技术,虽然网上资料多得很,但是质量参差不齐!恕在下愚昧,实在想不懂那些都不验证帖子里技术的正确性就直接转载的那些人,心里是怎么想的!你要是感觉帖子好,请默默的收藏到你的收藏夹里,等你确定帖子内容没错的时候你再转载好不好?你不知道你这样做,对这个技术的新手来说,无疑是灾难!你埋那么多坑,是怕别人学会了超越你吗?哎,言归正传,以下是我自己的学习总结,如果有哪里有错误、有问题,欢迎批评指...原创 2019-02-28 12:24:26 · 1273 阅读 · 0 评论 -
【Spark学习笔记】2、Java版-算子系列之filter(f:T => Boolean) : RDD[T] => RDD[T]
filter算子源码def filter(f: T => Boolean): RDD[T] = withScope { val cleanF = sc.clean(f) new MapPartitionsRDD[T, T]( this, (context, pid, iter) => iter.filter(cleanF), preservesPa...原创 2019-02-28 12:29:44 · 539 阅读 · 0 评论 -
【Spark学习笔记】3、Java版-算子系列之map(f:T => U) : RDD[T] => RDD[U]
map算子源码/** * Return a new RDD by applying a function to all elements of this RDD. */def map[U: ClassTag](f: T => U): RDD[U] = withScope { val cleanF = sc.clean(f) new MapPartitionsRDD[U, ...原创 2019-02-28 12:34:31 · 642 阅读 · 0 评论 -
【Spark学习笔记】4、Java版-算子系列之flatMap(f:T => Seq[U]) : RDD[T] => RDD[U]
flatMap源码/** * Return a new RDD by first applying a function to all elements of this * RDD, and then flattening the results. */def flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U] ...原创 2019-02-28 12:36:53 · 283 阅读 · 0 评论 -
【Spark学习笔记】5、Java版-算子系列之mapPartitions(f:Iterator[T] => Iterator[U]) : RDD[T] => RDD[U]
源码/** * Return a new RDD by applying a function to each partition of this RDD. * * `preservesPartitioning` indicates whether the input function preserves the partitioner, which * should be `fal...原创 2019-02-28 12:40:15 · 455 阅读 · 0 评论