Spark Sql 总结

1. 创建一个JavaSparkContext ：
    SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
JavaSparkContext sc = new JavaSparkContext(conf);

2. 创建rdd:
   (1) parallelize 一个Collection 
  List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
  JavaRDD<Integer> distData = sc.parallelize(data);
  
   (2) JavaRDD<String> distFile = sc.textFile("data.txt");
       读取data.txt的每一行内容
  
3.  RDD的方法。参考：http://blog.csdn.net/lxxc11/article/details/51333088


4.  创建一个SQLContext 
    JavaSparkContext sc = ...; // An existing JavaSparkContext.
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);

5.  创建DataFrame
(1) 
    DataFrame df = sqlContext.jsonFile("examples/src/main/resources/people.json");
(2)
DataFrame df = sqlContext.sql("select * from table ");

6.  将RDD转为DataFrame:
(1) 
JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
 new Function<String, Person>() {
public Person call(String line) throws Exception {
 String[] parts = line.split(",");


 Person person = new Person();
 person.setName(parts[0]);
 person.setAge(Integer.parseInt(parts[1].trim()));


 return person;
}
 });


DataFrame schemaPeople = sqlContext.createDataFrame(people, Person.class);


(2)
JavaRDD<String> people = sc.textFile("examples/src/main/resources/people.txt");
// The schema is encoded in a string
String schemaString = "name age";


// Generate the schema based on the string of schema
List<StructField> fields = new ArrayList<StructField>();
for (String fieldName: schemaString.split(" ")) {
 fields.add(DataType.createStructField(fieldName, DataType.StringType, true));
}
StructType schema = DataType.createStructType(fields);


// Convert records of the RDD (people) to Rows.
JavaRDD<Row> rowRDD = people.map(
 new Function<String, Row>() {
public Row call(String record) throws Exception {
 String[] fields = record.split(",");
 return Row.create(fields[0], fields[1].trim());
}
 });


// Apply the schema to the RDD.
DataFrame peopleDataFrame = sqlContext.createDataFrame(rowRDD, schema);


(3)
List<String> jsonData = Arrays.asList("{\"name\":\"Yin\",\"address\":{\"city\":\"Columbus\",\"state\":\"Ohio\"}}");
JavaRDD<String> anotherPeopleRDD = sc.parallelize(jsonData);
DataFrame anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD);