spark_hive 异常汇总

1.类型匹配失败
df.rdd.map({case Row(ab:String,bc:String)=>(ab,bc)}).collect().foreach(println(_))

18/08/02 11:39:37 INFO DAGScheduler: ResultStage 0 (collect at SDfd.scala:26) failed in 0.243 s due to Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): scala.MatchError: [null,null] (of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema)
    at com.mob.SDfd$$anonfun$main$1.apply(SDfd.scala:26)
    at com.mob.SDfd$$anonfun$main$1.apply(SDfd.scala:26)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)

【解决】

       df.rdd.map(row=>(row.getString(0),row.getString(1))).collect().foreach(println(_))

2. row不能被OrcInputFormat 序列化

      "spark.sql.orc.impl"="hive" :供两个参数（native/hive) native：使用spark提供OrcFileFormat序列化和反序列化orc文件，hive：使用hive提供的OrcInputFormat,OrcOutputFormat序列化和反序列化orc文件

      "spark.sql.orc.enableVectorizedReader"="true"