表分区出现损坏
1) Create a table"test". "create table test (n string) partitioned by (pstring)"
2) Load some data into partition(p='1')
3)Remove the path related to partition(p='1') of table test manually."hadoop fs -rmr ..../warehouse/..../test/p=1"
4)Run spark sql, spark-sql -e "select n from test where p='1';"
Then it throws exception:
Error:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:hdfs://hacluster/user/hive/warehouse/dp_tmp.db/spark_15044/p=2016-12-26(state=,code=0)
解决办法:
1、损坏分区数据不参与计算(hive采用这种方式解决)
setspark.sql.hive.verifyPartitionPath=true;
表示忽略这个损坏的分区数据
2、重新建分区overwrite 或者 drop tablepartition()