spark-1.6.3源码---启动spark shell报错

最新推荐文章于 2021-03-20 21:01:09 发布
最新推荐文章于 2021-03-20 21:01:09 发布
阅读量195 收藏
点赞数
分类专栏： Spark-1.6.3 文章标签： 源码 启动 bug

启动spark-shell报错:

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.3
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_192)
Type in expressions to have them evaluated.
Type :help for more information.
19/07/25 02:39:01 WARN Utils: Your hostname, <my-hostname> resolves to a loopback address: 127.0.1.1; using 172.99.5.154 instead (on interface eth0)
19/07/25 02:39:01 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/07/25 02:39:01 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:01 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:01 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:01 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:01 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
19/07/25 02:39:02 ERROR SparkContext: Error initializing SparkContext.
java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:748)
java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:748)

java.lang.NullPointerException
        at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1367)
        at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
        at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1033)
        at $iwC$$iwC.<init>(<console>:15)
        at $iwC.<init>(<console>:24)
        at <init>(<console>:26)
        at .<init>(<console>:30)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
        at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
        at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
        at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
        at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
        at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql
                ^

解决方案：

cp ${SPARK_HOME}/conf/spark-env.sh.template ${SPARK_HOME}/conf/spark-env.sh

# 修改spark-env.sh，添加
SPARK_LOCAL_IP=127.0.0.1
Ms_WangH
关注
专栏目录
Spark RPC 学习笔记
主要分享大数据相关的知识，如Spark、Hudi
07-17 226
我的原创地址：https://dongkelun.com/2020/12/23/sparkRPC/ 前言 学习记录一下Spark源码中的RPC，本文基于Spark3源码进行学习研究 三个重要的类 RpcEnv、RpcEndpoint、RpcEndpointRef主要是想搞懂这三个之间的关系 子类 Spark 1.6.3中默认使用了Netty作为底层的实现，但Akka的依赖依然存在；而Spark 2.1.0中的底层实现只有Netty，这样用户可以方便的使用不同版本的Akka或者将来某种更好的底层实现 Ne
Spark启动过程中遇到的错误 ./spark-shell –master spark://node001:7077
01-20
20/02/20 19:52:17 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.at.org.apache.spark.memory.Uni
spark-shell 启动失败，显示端口问题
weixin_30311605的博客
04-09 274
应该是端口问题，找不到spark节点。手动设置端口号 spark-shell --conf spark.shuffle.service.enabled=true --conf spark.replClassServer.port=15002 转载于:https://www.cnblogs.com/sunt9/p/6686004.html...
执行spark-shell报错：
weixin_30359021的博客
11-24 293
执行spark-shell报错： [ERROR] Terminal initialization failed; falling back to unsupportedjava.lang.NoClassDefFoundError: Could not initialize class scala.tools.fusesource_embedded.jansi.internal.Kernel32 ...
SparkShell 启动报错，在线等大神解决，急！！！！
Snow_Face的博客
03-01 1664
[root@hadoop01 bin]# ./spark-shell --master spark://hadooplog4j:WARN No appenders could be found for logger (org.apalog4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging...
单机运行 spark-shell错误
dickysun1987的专栏
12-18 3308
安装spark，spark-shell遇到错误
spark1.6.3:自己查看的spark1.6.3的源码，一些笔记记录等
03-23
阿帕奇火花 Spark是用于大数据的快速通用集群计算系统。 它提供了Scala，Java，Python和R中的高级API，以及优化的引擎，该引擎支持用于数据分析的通用...开始使用Spark的最简单方法是通过Scala shell： ./bin/spark
大数据(十一)--Spark概述
七宝.博客
11-18 2024
Spark概述Spark 历史Spark 简介 Spark 历史 Spark 简介
2021-03-19~20 大数据课程笔记 day58day59
qq_44745905的博客
03-20 592
@R星校长 1 Scala 1.1 【重点】Scala六大特性  1) 与java无缝整合  2) 类型自动推断  3) 并发和分布式  4) Trait特质特性  5) Match模式匹配  6) 高阶函数 1.2 【重点】Scala & Java异同？  1) 类型定义声明不同，Scala ：val/var Java强类型定义。  2) Scala中有类和对象，Java中有类。  3) Scala、java都是是基于Jvm,scala面向对象+面向函数编程，java面向对象编程。  4) .
大数据技术之_19_Spark学习_07_Spark 性能调优 + 数据倾斜调优 + 运行资源调优 + 程序开发调优 + Shuffle 调优 + GC 调优 + Spark 企业应用案例
黑泽君
05-04 2053
大数据技术之_19_Spark学习_07_Spark 性能调优 + 数据倾斜调优 + 运行资源调优 + 程序开发调优 + Shuffle 调优 + GC 调优 + Spark 企业应用案例
Spark-遇到的坑-启动spark报错
a1032818891的博客
07-26 1万+
1.解决措施： Please instead use:  - ./spark-submit with --num-executors to specify the number of executors  - Or set SPARK_EXECUTOR_INSTANCES  - spark.executor.instances to configure the number of instanc...
Spark学习中的问题汇总......
xiaozhou的博客
01-01 1684
第一次运行 run-example SparkPi时，Error spark.SparkContext:Error initializing SparkContext…错误 描述：在安装好Spark后，就尝试跑一下自带的示例程序： $ ./bin/run-example SparkPi 2&amp;amp;amp;gt;&amp;amp;amp;amp;1 | grep &amp;amp;quot;Pi is roughly&amp;amp;quot;
Spark运行shell报错：Service 'sparkDriver' could not bind on a random free port.
MyQGirl666的博客
05-16 9989
安装了spark之后，想要运行Scala版本的shell，于是进入Spark目录，执行下面的代码./bin/spark-shell出现报错，前面是一些WARN对于WARN WARN Utils: Service 'sparkDriver' could not bind on a random free port. You may check whether configuring an appro...
log4j:WARN no appenders could be found for logger
qq_38709565的博客
09-25 1042
初次使用ubuntu14.04+eclipse(mars)+hadoop-2.7.1开发环境调试程序时，选中main-》run as hadoop 提示如下警告： log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN...
log4j:WARN No appenders could be found for logger
热门推荐
Frank的博客
06-05 5万+
问题场景 在项目中使用Log4j时，出现以下情况： log4j:WARN No appenders could be found for logger (com.frank.Log4jDemo). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log...
Spark常见问题汇总
aiwode1234的专栏
05-16 1168
注意：如果Driver写好了代码，eclipse或者程序上传后，没有开始处理数据，或者快速结束任务，也没有在控制台中打印错误，那么请进入spark的web页面，查看一下你的任务，找到每个分区日志的stderr，查看是否有错误，一般情况下一旦驱动提交了，报错的情况只能在任务日志里面查看是否有错误情况了1、Operation category READ is not supported in state
单机运行Spark Shell遇到的一个低级错误
程裕强的专栏
04-08 2万+
bin/spark-shell下载spark-2.1.0-bin-hadoop2.7.tgz，解压缩直接进入spark根目录，然后运行bin/spark-shell即可进入。 但是今天遇到了一个低级错误： java.net.BindException: Cannot assign requested address: Service ‘sparkDriver’ failed after 16 r
Spark 部署及示例代码讲解
sdujava2011
01-26 800
Spark 部署 考虑到读者可能使用“ssh secure shell”这样的工具登陆 Linux，所以解释一下如何设置工具支持中文。 如何设置 ssh secure shell 支持中文 只需要设置下/etc/sysconfig/i18n 文件内容如清单 1 所示。 清单 1. 文件内容 LANG="zh_CN.GB18030" SUPPORTED="zh_CN.G
spark参数配置调优
提笔写忧伤
09-19 3万+
SPARK配置参数的两个地方： 1. $SPARK_HOME/conf/spark-env.sh  脚本上配置。 配置格式如下： export SPARK_DAEMON_MEMORY=1024m 2. 编程的方式（程序中在创建SparkContext之前，使用System.setProperty（“xx”，“xxx”）语句设置相应系统属性值)，即在spark-shell下配置 如：scal
linux搭建hive on spark环境(spark-1.6.3-without-hive+hadoop2.8.0+hive2.1.1)
最新发布
06-28
首先需要安装Hadoop和Spark，可以选择Hadoop 2.8.0和Spark 1.6.3版本。安装过程可以参考官方文档。 2. 安装Hive 安装Hive需要下载Hive 2.1.1版本，并解压到指定目录。然后需要配置Hive的环境变量，将Hive的bin目录...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值