关闭

Spark问题

标签: java
264人阅读 评论(0) 收藏 举报
分类:
15/07/09 17:50:36 INFO SecurityManager: Changing view acls to: dev001,
15/07/09 17:50:36 INFO SecurityManager: Changing modify acls to: dev001,
15/07/09 17:50:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dev001, ); users with modify permissions: Set(dev001, )
15/07/09 17:50:36 INFO Slf4jLogger: Slf4jLogger started
Exception in thread "main" org.jboss.netty.channel.ChannelException: Failed to bind to: /172.16.31.163:0
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
	at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391)
	at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388)
	at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
	at scala.util.Try$.apply(Try.scala:161)
	at scala.util.Success.map(Try.scala:206)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
	at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.BindException: 无法指定被请求的地址
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

相同问题:
  http://apache-spark-user-list.1001560.n3.nabble.com/org-jboss-netty-channel-ChannelException-Failed-to-bind-to-master-1xx-xx-xx-0-td8400.html
  stackoverflow上也有好多类似的



解决方法:

  1、某位外国人的结局方法 
The problem is resolved.I have added SPARK_LOCAL_IP=master in both slaves also.When i changed this my slaves are working.
Thank you all for your suggestions
2、好像确实需要把worker和master的spark-env.sh文件配置相同的。如下方式会把worker和master配置完全复制的 见如下: http://my.oschina.net/132722/blog/196027

然后在修改${SPARK_HOME}/conf下的slaves文件,添加上所有工作节点的主机名或者ip,注意是一行一个,这些工作节点服务器也需要配置ssh无密码登录,并加入master的公匙到工作节点。当然可以不配置,但是每次启动都输入密码很纠结。

然后使用scp复制spark文件到其他工作节点。

scp -r ${SPARK_HOME} hadoop@主机名:要复制到的目录
 
然后就可以在master主机${SPARK_HOME}/bin目录下运行start-all.sh启动集群



==========================================
这个问题主要是出现在使用master来提交算法时出现的,当使用worker机器提交时不会出现。
也就是说master提交有问题然而worker提交就会出现问题。





 
0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:60447次
    • 积分:1386
    • 等级:
    • 排名:千里之外
    • 原创:62篇
    • 转载:50篇
    • 译文:11篇
    • 评论:5条
    文章分类
    最新评论