先粘上wordCount代码:
val lines = sc.textFile()
val words = lines.flatMap(line => line.split(" "))
val pairs = words.map(word => (word, 1))
val counts = pairs.reduceByKey(_ + _)
counts.foreach(count => println(count._1 + ": " + count._2))
一、textFile()
sparkContext.scala:
/** * Read a text file from HDFS, a local file system (available on all nodes), or any * Hadoop-supported file system URI, and return it as an RDD of Strings. */ def textFile( path: String, minPartitions: Int = defaultMinPartitions): RDD[String] = withScope { assertNotStopped() hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text], minPartitions).map(pair => pair._2.toString).setName(path) }
def hadoopFile[K, V]( path: String, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minPartitions: Int = defaultMinPartitions): RDD[(K, V)] = withScope { assertNotStopped() // This is a hack to enforce loading hdfs-site.xml. // See SPARK-11227 for details. FileSystem.getLocal(hadoopConfiguration) // A Hadoop configuration can be about 10 KB, which is pretty big, so broadcast it. //对这个Hadoop配置信息进行广播 val confBroadcast = broadcast(new SerializableConfiguration(hadoopConfiguration)) val setInputPathsFunc = (jobConf: JobConf) => FileInputFormat.setInputPaths(jobConf, path) new HadoopRDD( this, confBroadcast, Some(setInputPathsFunc), inputFormatClass, keyClass, valueClass, minPartitions).setName(path) }
首先,hadoopFile方法的调用,会new 一个HadoopRDD,其中的元素是(key,value)pair,key是hdfs或文本文件的每一行的offset,value是文本行。然后对HadoopRDD调用了map方法(这个方法是在RDD.scala里面),剔除key,最终返回一个MapPartitionsRDD
为什么会说每个partition是hdfs的block,因为从HadoopRDD的getPartitions方法中可以看出,底层是调用的hadoop的inputFormat的getSplits方法
每个RDD最重要的就是compute()方法,通过这个方法拿到每个partition的元素
二、val words = lines.flatMap(line => line.split(" "))
flatMap方法又是创建一个MapPartitionsRDD
三、val pairs = words.map(word => (word, 1))
map方法也是创建一个MapPartitionsRDD
四、val counts = pairs.reduceByKey(_ + _)
在RDD中其实是找不到reduceByKey这个方法的,RDD中导入了隐式转换,implicit def
implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)]) (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairRDDFunctions[K, V] = { new PairRDDFunctions(rdd) }其实RDD里是没有reduceByKey的,因此对RDD调用reduceByKey()方法的时候,会触发scala的隐式转换;此时就会在作用域内,寻找隐式转换,会在RDD中找到rddToPairRDDFunctions()隐式转换,然后将RDD转换为PairRDDFunctions。
接着会调用PairRDDFunctions中的reduceByKey()方法
这里会返回一个kv的普通RDD
五、counts.foreach(count => println(count._1 + ": " + count._2))
之前一直说一个action会触发一个job,看完这个代码就明白了。
foreach会调用runJob方法
def runJob[T, U: ClassTag]( rdd: RDD[T], func: (TaskContext, Iterator[T]) => U, partitions: Seq[Int], resultHandler: (Int, U) => Unit): Unit = { if (stopped.get()) { throw new IllegalStateException("SparkContext has been shutdown") } val callSite = getCallSite val cleanedFunc = clean(func) logInfo("Starting job: " + callSite.shortForm) if (conf.getBoolean("spark.logLineage", false)) { logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString) } dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get) progressBar.foreach(_.finishAll()) rdd.doCheckpoint() }
在这个方法中调用了DAGScheduler的runJob方法。
后面再继续学习DAGScheduler的源码和运行机制。