15/07/09 17:50:36 INFO SecurityManager: Changing view acls to: dev001, 15/07/09 17:50:36 INFO SecurityManager: Changing modify acls to: dev001, 15/07/09 17:50:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dev001, ); users with modify permissions: Set(dev001, ) 15/07/09 17:50:36 INFO Slf4jLogger: Slf4jLogger started Exception in thread "main" org.jboss.netty.channel.ChannelException: Failed to bind to: /172.16.31.163:0 at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388) at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) at scala.util.Try$.apply(Try.scala:161) at scala.util.Success.map(Try.scala:206) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.net.BindException: 无法指定被请求的地址 at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:444) at sun.nio.ch.Net.bind(Net.java:436) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290) at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 相同问题: http://apache-spark-user-list.1001560.n3.nabble.com/org-jboss-netty-channel-ChannelException-Failed-to-bind-to-master-1xx-xx-xx-0-td8400.html stackoverflow上也有好多类似的 解决方法: 1、某位外国人的结局方法The problem is resolved.I have added SPARK_LOCAL_IP=master in both slaves also.When i changed this my slaves are working.Thank you all for your suggestions2、好像确实需要把worker和master的spark-env.sh文件配置相同的。如下方式会把worker和master配置完全复制的 见如下: http://my.oschina.net/132722/blog/196027然后在修改${SPARK_HOME}/conf下的slaves文件,添加上所有工作节点的主机名或者ip,注意是一行一个,这些工作节点服务器也需要配置ssh无密码登录,并加入master的公匙到工作节点。当然可以不配置,但是每次启动都输入密码很纠结。
然后使用scp复制spark文件到其他工作节点。
scp -r ${SPARK_HOME} hadoop@主机名:要复制到的目录
然后就可以在master主机${SPARK_HOME}/bin目录下运行start-all.sh启动集群
==========================================
这个问题主要是出现在使用master来提交算法时出现的,当使用worker机器提交时不会出现。
也就是说master提交有问题然而worker提交就会出现问题。
Spark问题
最新推荐文章于 2023-02-21 11:26:01 发布