Spark错误集锦(一)——spark.SparkContext: Created broadcast 0 from textFile at WordCount.scala:16

Spark错误集锦(一)——spark.SparkContext: Created broadcast 0 from textFile at WordCount.scala:16

yarn模式下运行spark提交任务：
Exception in thread “main” java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)

原因：hadoop集群采用Lzo压缩，Spark无法解析；
hadoop的core-site.xml 和mapred-site.xml中开启了压缩，并且压缩式lzo的。这就导致写入/上传到hdfs的文件自动被压缩为lzo了。

解决方法：spark-env.sh中增加配置如下：
`export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/module/hadoop-2.7.2/lib/native

export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar
`
提示：修改完配置后记得分发给集群！