spark整合kafka打包运行错误处理

问题1

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/serializer/StringDecoder
    at DirectKafkaWordCountDemo$.main(DirectKafkaWordCountDemo.scala:19)
    at DirectKafkaWordCountDemo.main(DirectKafkaWordCountDemo.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)

解决：

submit 时引入依赖包，此问题是缺少kafka_2.11-0.8.2.1.jar 和 metrics-core-2.2.0.jar

--jars /kafka-0-8_2.11-2.0.2.jar,/Users/yangyibo/.m2/repository/org/apache/kafka/kafka_2.11/0.8.2.1/kafka_2.11-0.8.2.1.jar,/Users/yangyibo/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar

---

问题2

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: com/yammer/metrics/Metrics
    at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85)
    at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26)
    at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:35)
    at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:46)

解决

submit 时引入依赖包，此问题是 metrics-core-2.2.0.jar

--jars /Users/yangyibo/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar