第一次在sparkR运行单词计数,遇到到下面问题。
16/08/09 11:42:07 ERROR RBackendHandler: collect on 40 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://nameservice1/user/root/user/yjl/aa.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd
程序默认是在hdfs的root路径下找文件的,这里是因为,没有指定好路径,所以文件找不到,正确的是:
lines <- SparkR:::textFile(sc,"/user/yjl/aa.txt")用"file///home/aa.txt"在集群本地找也是找不到的。