kylin 整合笔记

• 修改hadoop配置文件kylin访问异常
  1.修改hadoop配置文件重启hadoop服务
  2.重启完hadoop，需要重启hbase，个人经验要重启两次服务蔡靠谱，一次可能重启不成功。
  3.还要再挨个节点重启kylin。

• Directory /usr/local/hadoop/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
  执行那namenode format格式化：hdfs namenode -format

• 每次增量构建需要重启，否则找不到找不到配置文件
  修改工作节点kylin.properties文件

  kylin.server.mode=job
  kylin.server.cluster-servers=99.48.232.210:7070,99.48.232.211:7070,99.48.232.213:7070

• Caused by: org.apache.hadoop.hbase.TableExistsException: kylin_metadata
  这个错误是由于kylin已经运行过，元数据还保留着hbase中需要清理
  步骤如下：

  • hbase zkcli登录zk的客户端
  • ls /hbase/tablehbase的元信息在目录
  • rmr /hbase/table 删除元数据表

• ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
  这个原因主要是datanode unhealthy 在web界面可以看到。
  引起这个问题的原因为yarn-site.xml

    <property>
            <name>yarn.nodemanager.local-dirs</name>
            <value>/tmp/hadoop/nodemanager</value>
        </property>
        <property>
            <name>yarn.nodemanager.log-dirs</name>
            <value>/tmp/hadoop/nodemanager/logs</value>
        </property>

这两个配置路径磁盘空间不足，可以添加多磁盘，或者扩盘，或者清空这两个目录。

• File /kylin/kylin_metadata/kylin-f02df97d-cff7-4faf-8dc6-325ca5a738c5/ kylin_intermediate_kylin_error_log_cube_fde7b1bc_6249_47c1_a680_eb07e84772aa/_temporary/1/_temporary/ attempt_1532486903195_0015_m_000006_1/part-m-00006 could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.
  这个问题在我的实验环境中也是由yarn-site.xml

    <property>
            <name>yarn.nodemanager.local-dirs</name>
            <value>/tmp/hadoop/nodemanager</value>
        </property>
        <property>
            <name>yarn.nodemanager.log-dirs</name>
            <value>/tmp/hadoop/nodemanager/logs</value>
        </property>

这两个配置路径磁盘空间不足，可以添加多磁盘，或者扩盘，或者清空这两个目录。

- org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher Could not find any valid local directory for output/attempt_1532479393376_0050_r_000000_0/map_0.out 描述： Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#3 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1532479393376_0050_r_000000_0/map_0.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213) at org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput.(OnDiskMapOutput.java:65) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:265) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:514) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:336) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
原因：貌似与上一个问题类似原因