sparkSQL写数据到hdfs中出现Snappy报错

今天spark在写数据到hive中出现错误


Job aborted due to stage failure: Task 7 in stage 5.0 failed 4 times, most recent failure: Lost task 7.3 in stage 5.0 (TID 68,
NM-304-SA5212M4-BIGDATA-2016-356.BIGDATA.CHINATELECOM.CN):
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
at org.apache.hadoop.io.compress.Codec.checkNativeCodeLoaded(SnappyCodec.java:63)
at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)
at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:98)
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:136)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1191)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
乍看之下，是Snappy压缩的问题。但是，在/etc/spark/conf下面的spark-default.conf里面已经配置了
extraClassPath
spark.driver.extraClassPath /usr/lib/hadoop/lib/*:/usr/lib/spark/hlib/*
spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native/
spark.executor.extraClassPath /usr/lib/hadoop/lib/
spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native/*
但是问题就出在这里，最后一行有个*号。
导致Snappy jar包重复，出现
可以看到，有个duplicate的字样，说明有上传重复的的文件。去掉*号，正常运行，不再报错