Spark mmlib开发中遇到的问题

1.问题1 SparkContext初始化失败，master URL 没找到
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
17/03/01 20:22:00 INFO SparkContext: Running Spark version 1.6.1
17/03/01 20:22:00 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.(SparkContext.scala:401)
at com.jd.data.center.machine.learn.train.WordCount$.main(WordCount.scala:12) at com.jd.data.center.machine.learn.train.WordCount.main(WordCount.scala) 17/03/01 20:22:00 INFO SparkContext: Successfully stopped SparkContext Exception in thread “main” org.apache.spark.SparkException: A master URL must be set in your configuration at org.apache.spark.SparkContext.(SparkContext.scala:401) at com.jd.data.center.machine.learn.train.WordCount$.main(WordCount.scala:12)
at com.jd.data.center.machine.learn.train.WordCount.main(WordCount.scala)

解决办法：
方法一：在configuration中vm参数中设置： -Dspark.master=local即可
方法二： val conf = new SparkConf().setMaster(“local”).setAppName(“WordCount”)即在SparkConf实例后设置其Master，local就是本机。上线后，不能设置该项为local。

2.Failed to locate the winutils binary in the hadoop binary path
由于本机没有安装hadoop导致的。因此，下载hadoop的windows版本，http://www.srccodes.com/p/article/39/error-util-shell-failed-locate-winutils-binary-hadoop-binary-path，
ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
If we directly take the binary distribution of Apache Hadoop 2.2.0 release and try to run it on Microsoft Windows, then we’ll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path.In the previous post - Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS, I have already described how to build Windows distribution of Apache Hadoop 2.2.0. But if you are feeling little bit lazy to perform all the lengthy steps described there and want to get started with Hadoop quickly by-passing those steps, then this is the post worth looking into.
原因：The binary distribution of Apache Hadoop 2.2.0 release does not contain some windows native components (like winutils.exe, hadoop.dll etc). These are required (not optional) to run Hadoop on Windows.
Tools and Technologies used in this article :
Apache Hadoop 2.2.0
Windows 7 OS
JDK 1.6