SparkSQL使用的简单例子

• 从json文件创建dataFrame

 val df: DataFrame = sqlContext.read.json("hdfs://master:9000/user/spark/data/people.json")
 val people = df.registerTempTable("person")
 val teenegers: DataFrame = sqlContext.sql("select name,age from person")
 teenegers.map(x = "name:" + x(0)+ " " + "age:" + x(1)).collect().foreach(println)

• 从parquet文件创建dataFrame

 val df: DataFrame = sqlContext.read.parquet("hdfs://master:9000/user/spark/data/namesAndAges.parquet")
 val people = df.registerTempTable("person")
 val teenegers: DataFrame = sqlContext.sql("select name,age from person")
 teenegers.map(x = "name:" + x(0)+ " " + "age:" + x(1)).collect().foreach(println)

• 从普通RDD创建dataFrame_1

 val people = sc.textFile("hdfs://master:9000/user/spark/data/people.txt")
        .map(_.split(",")).map(p = Person(p(0), p(1).trim.toInt))