关闭

Spark Sql 总结

148人阅读 评论(0) 收藏 举报
分类:
1. 创建一个JavaSparkContext :
    SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
JavaSparkContext sc = new JavaSparkContext(conf);

2. 创建rdd:
   (1) parallelize 一个Collection 
  List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
  JavaRDD<Integer> distData = sc.parallelize(data);
  
   (2) JavaRDD<String> distFile = sc.textFile("data.txt");
       读取data.txt的每一行内容
  
3.  RDD的方法。参考:http://blog.csdn.net/lxxc11/article/details/51333088


4.  创建一个SQLContext 
    JavaSparkContext sc = ...; // An existing JavaSparkContext.
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);

5.  创建DataFrame
(1) 
    DataFrame df = sqlContext.jsonFile("examples/src/main/resources/people.json");
(2)
DataFrame df = sqlContext.sql("select * from table ");

6.  将RDD转为DataFrame:
(1) 
JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
 new Function<String, Person>() {
public Person call(String line) throws Exception {
 String[] parts = line.split(",");


 Person person = new Person();
 person.setName(parts[0]);
 person.setAge(Integer.parseInt(parts[1].trim()));


 return person;
}
 });


DataFrame schemaPeople = sqlContext.createDataFrame(people, Person.class);


(2)
JavaRDD<String> people = sc.textFile("examples/src/main/resources/people.txt");
// The schema is encoded in a string
String schemaString = "name age";


// Generate the schema based on the string of schema
List<StructField> fields = new ArrayList<StructField>();
for (String fieldName: schemaString.split(" ")) {
 fields.add(DataType.createStructField(fieldName, DataType.StringType, true));
}
StructType schema = DataType.createStructType(fields);


// Convert records of the RDD (people) to Rows.
JavaRDD<Row> rowRDD = people.map(
 new Function<String, Row>() {
public Row call(String record) throws Exception {
 String[] fields = record.split(",");
 return Row.create(fields[0], fields[1].trim());
}
 });


// Apply the schema to the RDD.
DataFrame peopleDataFrame = sqlContext.createDataFrame(rowRDD, schema);


(3)
List<String> jsonData = Arrays.asList("{\"name\":\"Yin\",\"address\":{\"city\":\"Columbus\",\"state\":\"Ohio\"}}");
JavaRDD<String> anotherPeopleRDD = sc.parallelize(jsonData);
DataFrame anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD);




   
0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:42074次
    • 积分:846
    • 等级:
    • 排名:千里之外
    • 原创:43篇
    • 转载:6篇
    • 译文:0篇
    • 评论:8条
    文章分类
    最新评论