用spark-shell命令时候报错了——aboutyun

最新推荐文章于 2024-06-16 13:44:00 发布

木生火18624

最新推荐文章于 2024-06-16 13:44:00 发布

阅读量780

点赞数 1

分类专栏：大数据Spark

本文链接：https://blog.csdn.net/penghao_1/article/details/103801077

版权

在尝试使用`spark-shell`时遇到了两个错误。首先，由于`/data/spark_data/history/event-log`目录不存在，导致了FileNotFoundException。然后，尽管创建了该目录，但启动`spark-shell`时出现了Permission denied的错误。问题在于目录或其父目录没有给予足够的读写权限。解决方法是对`data`目录进行777权限设置，如`chmod 777 data`，确保有权限写入。完成授权后，再次运行`spark-shell`命令。

摘要由CSDN通过智能技术生成

[aboutyun@master ~]$ spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
20/01/02 10:42:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/01/02 10:42:23 ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/data/spark_data/history/event-log does not exist
   at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
   at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
   at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
   at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:93)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
   at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
   at scala.Option.getOrElse(Option.scala:121)
   at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
   at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
   at $line3.$read$$iw$$iw.<init>(<console>:15)
   at $line3.$read$$iw.<init>(<console>:31)
   at $line3.$read.<init>(<console>:33)
   at $line3.$read$.<init>(<console>:37)
   at $line3.$read$.<clinit>(<console>)
   at $line3.$eval$.$print$lzycompute(<console>:7)
   at $line3.$eval$.$print(<console>:6)
   at $line3.$eval.$print(<console>)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
   at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
   at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
   at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
   at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
   at scala.reflect.internal.util.AbstractFileClassLoa