Spark机器学习之第一弹你懂的踩坑记录！

Spark机器学习之踩第一个坑！

A signature in package.class refers to type compileTimeOnly in package scala.annotation which is not available.错误解决

/**
  * Created by danger on 2016/11/19.
  * 使用SGD算法逻辑回归的垃圾邮件分类器
  */
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.classification.LogisticRegressionWithSGD
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.regression.LabeledPoint
object AntiSpam {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName(s"MLlib example").setMaster("local[5]")
     val sc = new SparkContext(conf)

    // Load 2 types of emails from text files: spam and ham (non-spam).
     // Each line has text from one email.
     val spam = sc.textFile("data\\spam.txt")
     val ham = sc.textFile("data\\ham.txt")

     // Create a HashingTF instance to map email text to vectors of 100 features.
     val tf = new HashingTF(numFeatures = 100)
     // Each email is split into words, and each word is mapped to one feature.
     val spamFeatures = spam.map(email => tf.transform(email.split(" ")))
     val hamFeatures = ham.map(email => tf.transform(email.split(" ")))

     // Create LabeledPoint datasets for positive (spam) and negative (ham) examples.
     val positiveExamples = spamFeatures.map(features => LabeledPoint(1, features))
     val negativeExamples = hamFeatures.map(features => LabeledPoint(0, features))
     val trainingData = positiveExamples ++ negativeExamples
     trainingData.cache() // Cache data since Logistic Regression is an iterative algorithm.

    // Create a Logistic Regression learner which uses the SGD.
     val lrLearner = new LogisticRegressionWithSGD()
     // Run the actual learning algorithm on the training data.
     val model = lrLearner.run(trainingData)

     // Test on a positive example (spam) and a negative one (ham).
     // First apply the same HashingTF feature transformation used on the training data.
     val posTestExample = tf.transform("O M G GET cheap stuff by sending money to ...".split(" "))
     val negTestExample = tf.transform("Hi Dad, I started studying Spark the other ...".split(" "))
     // Now use the learned model to predict spam/ham for new emails.
     println(s"Prediction for positive test example: ${model.predict(posTestExample)}")
     println(s"Prediction for negative test example: ${model.predict(negTestExample)}")

     sc.stop()
  }

}

Error:scalac: bad symbolic reference. A signature in package.class refers to type compileTimeOnly
in package scala.annotation which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling package.class.

换成2.11后，没有编译错误，但是运行的时候：

16/11/19 23:06:43 INFO SparkContext: Created broadcast 1 from textFile at AntiSpam.scala:19
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:224)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2400)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2411)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
	at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:653)

果断删除pom.xml中过多的jar包。

Prediction for positive test example: 1.0
Prediction for negative test example: 0.0