win10启动hadoop时遇到的坑

目录

背景

1、 Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.

2、 Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory D:\tmp\hadoop-123\dfs\name is in an inconsistent state: storage directory does not exist or is not accessible.

3、 Exiting with status 1: java.io.IOException: All specified directories are not accessible or do not exist.

4、NameNode is not formatted.

1、URI has an authority component

2、 util.ExitUtil: Exiting with status 1: java.lang.NullPointerException

5、Error occurred during initialization of VM

启动Hadoop

---

背景

昨日编译好了hadoop3.0.2源码，今日启动之，孰料想，又出现一堆错误

 

1、 Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.

     启动不了nameNode，解决方法：

 

在hadoop文件夹\hadoop-3.0.2\etc\hadoop\core-site.xml的<configuration>标签下，添加如下内容

<property>

 

    <name>fs.default.name</name>

 

    <value>hdfs://localhost:9000</value>

 

  </property>

 

2、 Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory D:\tmp\hadoop-123\dfs\name is in an inconsistent state: storage directory does not exist or is not accessible.

      dfs\name文件夹没有创建。解决方法：

     在某处创建dfs文件夹，然后在dfs文件夹里创建name和data文件夹，并且在hadoop文件夹\etc\hadoop\core-site.xml的<configuration>标签下，添加如下内容：

     <property>

        <name>hadoop.tmp.dir</name>

        <value>D:/ProgramCoding/hadoop-3.0.2/workspace</value>

    </property>

 

    <property>

        <name>dfs.data.dir</name>

            <value>D:/ProgramCoding/hadoop-3.0.2/workspace/dfs/data</value>

        </property>

    <property>

        <name>dfs.name.dir</name>

        <value>D:/ProgramCoding/hadoop-3.0.2/workspace/dfs/name</value>

 

3、 Exiting with status 1: java.io.IOException: All specified directories are not accessible or do not exist.

     解决方法：

     检查core_site.xml文件里所有文件分隔符，必须是/而不是windows的\

 

4、NameNode is not formatted.

     解决方法：

     在cmd里运行命令hadoop namenode -format格式化namenode

格式化namenode时，又遇到一些问题

1、URI has an authority component

     

     打开hadoop文件夹\etc\hadoop\hdfs-site.xml，在<configuration>标签里，输入如下内容

         <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/D:/ProgramCoding/hadoop-3.0.2/workspace/dfs/namenode/name</value>

         </property>

 

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:/D:/ProgramCoding/hadoop-3.0.2/workspace/dfs/namenode/data</value>

    </property>

前提是保证两个<value>标签下的文件夹都存在，如不存在，就创建

 

2、 util.ExitUtil: Exiting with status 1: java.lang.NullPointerException

java.lang.NullPointerException

at java.io.File.(File.java:277)

at org.apache.hadoop.hdfs.server.namenode.NNStorage.setStorageDirectories(NNStorage.java:300)

at org.apache.hadoop.hdfs.server.namenode.NNStorage.(NNStorage.java:161)

at org.apache.hadoop.hdfs.server.namenode.FSImage.(FSImage.java:127)

at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:829)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)

 

     解决方法：检查hadoop文件夹\hadoop-3.0.2\etc\hadoop\hdfs-site.xml文件里<value>的值，绝对路径前面必须加上file:/

 

解决完这两个问题后，就可以正常格式化namenode了

5、Error occurred during initialization of VM

Could not reserve enough space for 1024000KB object heap

 

     内存预留过大问题，解决方法：

     修改文件hadoop文件夹\etc\hadoop\hadoop-env.cmd，添加或修改HADOOP_HEAPSIZE的值为200（默认1000）

启动Hadoop

解决完上面问题后，就可以start-all了

出现四个窗口，命令运行不结束就行