1.类型匹配失败 df.rdd.map({case Row(ab:String,bc:String)=>(ab,bc)}).collect().foreach(println(_)) 18/08/02 11:39:37 INFO DAGScheduler: ResultStage 0 (collect at SDfd.scala:26) failed in 0.243 s due to Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): scala.MatchError: [null,null] (of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema) at com.mob.SDfd$$anonfun$main$1.apply(SDfd.scala:26) at com.mob.SDfd$$anonfun$main$1.apply(SDfd.scala:26) at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) at scala.collection.Iterator$class.foreach(Iterator.scala:742)
【解决】
df.rdd.map(row=>(row.getString(0),row.getString(1))).collect().foreach(println(_))
2. row不能被OrcInputFormat 序列化
"spark.sql.orc.impl"="hive" :供两个参数(native/hive) native:使用spark提供OrcFileFormat序列化和反序列化orc文件,hive:使用hive提供的OrcInputFormat,OrcOutputFormat序列化和反序列化orc文件
"spark.sql.orc.enableVectorizedReader"="true"