flatmap
flatmap 简单用法
def main ( args: Array[ String] ) : Unit = {
Logger. getLogger ( "org" ) . setLevel ( Level. WARN)
val conf = new SparkConf ( ) . setAppName ( "WordCountDemo" ) . setMaster ( "local" )
val sc = new SparkContext ( conf)
sc. textFile ( "hdfs://hadoop001:9000/in/word" )
. flatMap ( _. split ( " " ) ) . map ( ( _, 1 ) ) . reduceByKey ( _ + _) . collect ( ) . foreach ( println ( _) )
sc. stop ( )
}
flatmap 复杂用法
def main ( args: Array[ String] ) : Unit = {
val spark = SparkSession. builder ( )
. config ( "spark.executor.memory" , "2g" )
. appName ( "WordCount" )
. master ( "local[6]" )
. getOrCreate ( )
val sc = spark. sparkContext
val array = Array ( ( 1 , "a b c d" ) , ( 2 , "a b c" ) , ( 3 , "a b" ) )
val rdd = sc. parallelize ( array)
rdd. flatMap ( x = > {
val sub = ArrayBuffer. empty[ ( Int, String) ]
x. _2. split ( " " ) . foreach ( e = > {
sub. += ( ( x. _1, e) )
} )
sub. iterator
} ) . collect ( ) . foreach ( println)
sc. stop ( )
spark. close ( )
}
结果
(1,a)
(1,b)
(1,c)
(1,d)
(2,a)
(2,b)
(2,c)
(3,a)
(3,b)
一块石头一半做成了佛像,一半做成了台阶。台阶不服气的问佛:我们本是一块石头,凭什么人们都踩着我,而去朝拜你呢?佛说: 因为你只挨了一刀,而我却经历了千刀万剐,千锤万凿。台阶沉默了。第二天台阶又来找佛,说:这是我的朋友切菜板,您给个说法?