Apache Spark DataFrames入门指南：创建DataFrame

Apache Spark DataFrames入门指南：创建DataFrame
http://www.iteblog.com/archives/1565(转

1:从csv文件创建DataFrame
1/在build.sbt文件里面田间spark-csv支持库
<dependency>
    <groupid>com.databricks</groupid>
    <artifactid>spark-csv_2.10</artifactid>
    <version>1.3.0</version>
</dependency>
3/
import org.apache.spark.SparkConf
val conf = new SparkConf().setAppName("csvDataFrame").setMaster("local[2]")
4/
val sc = new SparkContext(conf) 
5、
val sqlContext=new SQLContext(sc) 
6
import com.databricks.spark.csv._
val students=sqlContext.csvFile(filePath="StudentData.csv", useHeader=true, delimiter='|') 


val options = Map("header" -> "true", "path" -> "E:\\StudentData.csv")
val newStudents = sqlContext.read.options(options).format("com.databricks.spark.csv").load()


7 数据
id|studentName|phone|email
1|Burke|1-300-746-8446|ullamcorper.velit.in@ametnullaDonec.co.uk
2|Kamal|1-668-571-5046|pede.Suspendisse@interdumenim.edu
3|Olga|1-956-311-1686|Aenean.eget.metus@dictumcursusNunc.edu
4|Belle|1-246-894-6340|vitae.aliquet.nec@neque.co.uk
5|Trevor|1-300-527-4967|dapibus.id@acturpisegestas.net
6|Laurel|1-691-379-9921|adipiscing@consectetueripsum.edu
7|Sara|1-608-140-1995|Donec.nibh@enimEtiamimperdiet.edu
8|Kaseem|1-881-586-2689|cursus.et.magna@euismod.org
9|Lev|1-916-367-5608|Vivamus.nisi@ipsumdolor.com
10|Maya|1-271-683-2698|accumsan.convallis@ornarelectusjusto.edu
11|Emi|1-467-270-1337|est@nunc.com
12|Caleb|1-683-212-0896|Suspendisse@Quisque.edu
13|Florence|1-603-575-2444|sit.amet.dapibus@lacusAliquamrutrum.ca
14|Anika|1-856-828-7883|euismod@ligulaelit.co.uk
15|Tarik|1-398-171-2268|turpis@felisorci.com
16|Amena|1-878-250-3129|lorem.luctus.ut@scelerisque.com
17|Blossom|1-154-406-9596|Nunc.commodo.auctor@eratSed.co.uk
18|Guy|1-869-521-3230|senectus.et.netus@lectusrutrum.com
19|Malachi|1-608-637-2772|Proin.mi.Aliquam@estarcu.net
20|Edward|1-711-710-6552|lectus@aliquetlibero.co.uk