Error: java.io.IOException: Failed to create local dir in /tmp/blockmgr-0937276e-0a89-4df0-9282-724a0525363a/1e. (state=,code=0)
java.io.IOException: Failed to create local dir in /tmp/spark-cb77f580-1ba5-47be-929c-7ed1ce74baf4/executor-00b54518-84f6-4aa1-a711-b077c67c9ac2/blockmgr-db8215f4-cb59-4a34-ac1c-ca9e435367cc/34.
初步怀疑跟map输出临时文件有关系,但又未在/tmp下找到那种大量的临时目录文件,最后暂靠重启服务解决。
2、spark thriftserver执行以下sql,不同时间段出现不同task数,9点钟执行竟然产生近3万个左右(导致初始化时间花了5分钟,实际执行花了20s),后面再执行又恢复正常(执行时间1s内),一直维持在406个task。
insert overwrite table dim_theme_category_xjp select * from dim_theme_category
3、NM出现大量container自杀情况,尚未找到原因:
2017-10-21 02:31:29,111 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e118_1506579421178_160745_01_000013
2017-10-21 02:31:29,111 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop IP=10.51.20.155 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1506579421178_160745 CONTAINERID=container_e118_1506579421178_160745_01_000013
2017-10-21 02:31:29,112 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e118_1506579421178_160745_01_000013 transitioned from RUNNING to KILLING
2017-10-21 02:31:32,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 15988 for container-id container_e118_1506579421178_160739_01_000001: 283.2 MB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used
2017-10-21 02:31:32,097 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e118_1506579421178_160746_01_000022 is : 1
2017-10-21 02:31:32,097 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_e118_1506579421178_160746_01_000022 and exit code: 1
以上日志可以看到容器分配了1g物理内存,2.1g虚拟内存,只使用了283M PM, 1.5G vm,内存有剩余但仍被杀掉,网上查资料说可能是map/reduce中永久代内存不足的问题(默认为64m),但没找到具体解释。
4、spark ui job/stage一直停留在active状态,永远不会结束:
spark在大量并发情况下,容易出现内存泄露,导致部分sql在spark ui中一直处于active状态,而实际查看日志是运行成功的。
且暂时没找到办法来调整一直处于active状态的job,ui上也没办法杀。目前没找到合适办法解决该问题。
5、基于spark thrift 的insert权限问题:
当指定一个用户执行insert操作时(beeline -u "jdbc:hive2://bis-newdatanode-s2b-80:10050" -n gnbidev -p ''),报权限问题
insert into test_data_tmp7 select * from test_data_tmp1;
提示以下问题:
Error: org.apache.hadoop.security.AccessControlException: Permission denied: user=gnbidev, access=WRITE, inode="/user/hive/warehouse/test.db/test_data_tmp7/.hive-staging_hive_2017-11-29_15-10-21_297_733955344912498349-1/-ext-10000/_temporary/0/task_201711291510_0000_m_000000/part-00000":hadoop:supergroup:drwxr-xr-x
通过查看test_data_tmp7表文件,发现有一部分文件归属到hadoop账号下,从而导致权限问题