配置好spark standalone集群模式后,执行脚本命令
sbin/start-all.sh
查看进程
jpsall
=============== hadoop100 ===============
31009 Worker
25924 NameNode
38165 Jps
30824 Master
27179 JobHistoryServer
26108 DataNode
28092 QuorumPeerMain
26829 NodeManager
=============== hadoop102 ===============
43045 Jps
37000 Worker
34746 QuorumPeerMain
27755 ResourceManager
27915 NodeManager
27277 DataNode
=============== hadoop103 ===============
103987 NodeManager
26402 Worker
24357 QuorumPeerMain
103479 DataNode
32012 Jps
103692 SecondaryNameNode
可见master和worker进程已经在集群上起来了,但是访问web页面http://hadoop100:8080时出现以下页面:
查看端口,发现已经被占用:
[techd@hadoop100 sbin]$ lsof -i tcp:8080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 99187 techd 49u IPv6 184141275 0t0 TCP *:webcache (LISTEN)
于是顺藤摸瓜,找到该PID对应的进程,发现竟然是zookeeper进程。原来是zookeeper默认客户端端口号也是8080,与spark端口起了冲突。因此修改zookeeper目录下的conf/zoo.cfg文件,在最后一行添加:
admin.serverPort=9099
给zookeeper随意定义一个不冲突的端口号即可