YARN
文章平均质量分 59
源神
天道酬勤
展开
-
YARN启动ResourceManager报错:Operation category READ is not supported in state standby
NameNode启用了HA,发现Yarn的ResourceManage启动不了,查看日志:2018-02-28 09:47:27,059 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1232)) - Error starting ResourceManagerorg.apache.hadoop.service.S...原创 2018-02-28 10:14:34 · 1596 阅读 · 0 评论 -
YARN的App Timeline Server无法启动
查看日志,报错:too many open file(1)设置打开文件的最大数值# ulimit -a# ulimit -n 65535# vim /etc/security/limits.conf新增内容:* soft nofile 65535* hard nofile 65535(2)echo 655360 > /proc/sys/fs/file-max编辑文件/etc/sysctl.c...原创 2018-03-02 18:42:27 · 2609 阅读 · 1 评论 -
Yarn配置标签的调度策略,刷新队列报错: default cannot be found during refresh
refreshQueues: java.io.IOException: Failed to re-init queues at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.l...原创 2018-02-24 16:09:21 · 2219 阅读 · 0 评论 -
Ambari—YARN配置ResourceManager HA(二)
2、ResourceManager2.1 先决条件(1)群集中必须至少有三台主机(2)运行Apache ZooKeeper服务器至少三台2.2 操作步骤(1)在Web主界面中,浏览 Services > YARN > Summary(2)选择Service Actions并选择Enable ResourceManager HA。"Enable ResourceManager HA"向导...原创 2018-03-05 16:17:54 · 4354 阅读 · 0 评论 -
Yarn启用HA后,删除定义的配置参数,如:yarn.resourcemanager.hostname
Yarn启用HA后,由于yarn.resourcemanager.hostname导致spark提交运行不了,但ambari界面无法删除配置参数。手动删除办法:(1)Ambari Web界面停止YARN和Zookeeper;(2)在Ambari Server主机上调用如下API,获取yarn-site.json文件/var/lib/ambari-server/resources/scripts/c...原创 2018-03-26 09:33:18 · 3257 阅读 · 0 评论 -
nodemanager启动:Failed to initialize spark2_shuffle...java.net.BindException: Address already in use
提示端口被占用,但日志没找到具体端口,默认spark.shuffle.service.port是7337,日志有打印但发现未被占用然后找了下yarn和spark配置文件相关的端口,还列出所有端口,一个个根据端口找到应用排查,折腾了半个多小时,最终还是采用重启大法搞定...原创 2018-04-09 10:25:18 · 1550 阅读 · 0 评论 -
启用Kerberos后,Yarn中ResourceManager启动失败:KeeperErrorCode = BadVersion for /yarn-leader-election
日志报错:2018-08-09 11:36:21,783 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService failed in state INIT...原创 2018-08-09 11:45:25 · 3468 阅读 · 1 评论 -
Yarn升级报:retries get failed due to exceeded maximum allowed retries number: 0
日志:18/08/17 09:11:22 WARN ipc.Client: Failed to connect to server: 133.bd/**.**.**.**:8032: retries get failed due to exceeded maximum allowed retries number: 0java.net.ConnectException: 拒绝连接 a...原创 2018-08-17 09:21:19 · 6246 阅读 · 1 评论