Spark任务都会绑定一个端口来显示WebUI,默认端口为4040,如果被占用则依次递增+1端口重试,重试次数由参数spark.port.maxRetries=16控制,默认重试16次后就放弃执行
当有17个app进来会报错,最多能运行16个作业
我们工作中,一定要把这个参数调大,一般都好几千
spark.eventLog.enabled=true 即使spark任务运行结束,也可以访问sparkUI,来保证任务跑完了也可以实时监控
./sbin/start-history-server.sh
并把日志存到文件系统里面去
spark.port.maxRetries | 16 | Maximum number of retries when binding to a port before giving up. When a port is given a specific value (non 0), each subsequent retry will increment the port used in the previous attempt by 1 before retrying. This essentially allows it to try a range of ports from the start port specified to port + maxRetries. |
1. 程序代码中初始化SparkConf时,设置conf.set(“spark.port.maxRetries”,“128”)
2. 使用spark-submit提交任务时,--conf spark.port.maxRetries=128
3. 在全局的spark-defaults.conf中添加spark.port.maxRetries 128 , 对所有的application起作用