数据之路

User [dr.who] is not authorized to view the logs for application 原因 Resource Manager UI的默认用户dr.who权限不正确

在spark on yarn运行中，有时会发现spark程序运行完毕后，spark的运行界面没有信息，或者找不到相关的运行信息了，经仔细查看NodeManager UI,出现如下信息：Log Aggregation Status TIME_OUT原来NodeManager可以在应用结束后将日志安全地移动到分布式文件系统HDFS，当应用(application)结束时，用户能通过 YARN 的命令行

dfs.datanode.du.reserved 预留空间不生效的问题

修改的ranger ui的admin用户登录密码时，需要在ranger的配置里把admin_password改成一样的，否则hdfs的namenode在使用admin时启动不起来,异常如下:Traceback (most recent call last): ambari_ranger_admin, ambari_ranger_password = self.create_ambari_admin_user(ambari_ranger_admin, ambari_ranger_password, f

Cannot obtain block length for LocatedBlock

异常信息如下：  2015-12-09 17:39:20,310 ERROR datanode.DataNode (DataXceiver.java:run(278)) - hadoop07:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:36479 dst: /127.0.0.1:50010

java.lang.OutOfMemoryError: PermGen space at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineCl

前段时间公司hadoop集群宕机，发现是namenode磁盘满了, 清理出部分空间后，重启集群时，重启失败。 又发现集群Secondary namenode 服务也恰恰坏掉，导致所有的操作log持续写入edits.new 文件，等集群宕机的时候文件大小已经达到了丧心病狂的70G+..重启集群报错 加载edits文件失败。分析加载文件报错原因是磁盘不足导致最后写入的log只写入一半就宕机了。由

Datanode的日志中看到: 10/12/14 20:10:31 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get ne

480000 millis timeout while waiting for channel to be ready for write

Hbase依赖的datanode日志中如果出现如下报错信息：DataXceiverjava.io.EOFException: INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block  解决办法：Hbase侧配置的dfs.socket.tim

2014-08-24 22:15:21,714 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing datanode Command java.io.IOException: Error in deleting blocks.         at org.apache.hadoop.hdfs.serve

Request received to kill task 'attempt_201411191723_2827635_r_000009_0' by user ------- Task has been KILLED_UNCLEAN by the user 1.An impatient user (armed with "mapred job -kill-task" command)

org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://nameservice/user/hive/warehouse/om_dw.db/mac_wifi_day_data/tid=CYJOY/.hive-staging_hive_2016-01-20_10-19-09_200_1

AM Container for appattempt_1453292851883_0381_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://hadoop:8088/cluster/app/application_1453292851883_01

一个测试环境hadoop集群由于磁盘满导致宕机，启动后发现journalnode报如下异常：2018-03-19 20:48:04,817 WARN  namenode.FSImage (EditLogFileInputStream.java:scanEditLog(359)) - Caught exception after scanning through 0 ops from /data1_...