解决 Windows.createDirectoryWithMode0

解决Windows.createDirectoryWithMode0

最近在学习Hadoop ，在window本地运行Hadoop时，老是出现 java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V ，通过自己的各种尝试，终于解决了这个问题。

Hadoop安装

开发环境：Win10；
Hadoop版本：hadoop-2.6.0-cdh5.16.2.tar
安装步骤

1. 以管理员身份启动WinRAR 解压 hadoop-2.6.0-cdh5.16.2.tar 解压到指定目录，我的安装目录是： D:\WorkTools\Hadoo;务必以管理员身份运行，否则在解压的过程中会出现错误。
2. 配置环境变量：HADOOP_HOME=D:\WorkTools\Hadoop，path=%HADOOP_HOME%/bin；
3. 配置%HADOOP_HOME%\etc\hadoop\core-site.xml；

   <configuration>
   	<property>
   		<name>fs.default.name</name>
   		<value>hdfs://localhost:8020</value>
   	</property>
   </configuration>
   

4. 配置%HADOOP_HOME%\etc\hadoop\hdfs-site.xml；

   <configuration> 
       <property>  
           <name>dfs.replication</name>  
           <value>1</value>  
       </property>  
       <property>   
        <name>dfs.permissions</name>   
        <value>false</value>   
     </property>  
      <property>  
          <name>dfs.namenode.name.dir</name>  
          <value>/E:/WorkSpaces/Repositories/Hadoop/Data/NameNode</value>  
      </property>  
      <property>  
           <name>fs.checkpoint.dir</name>  
           <value>/E:/WorkSpaces/Repositories/Hadoop/Data/snn</value>  
       </property>  
       <property>  
           <name>fs.checkpoint.edits.dir</name>  
           <value>/E:/WorkSpaces/Repositories/Hadoop/Data/snn</value>  
       </property>  
          <property>  
          <name>dfs.datanode.data.dir</name>  
          <value>/E:/WorkSpaces/Repositories/Hadoop/Data/DataNode</value>  
      </property>  
   </configuration>
   

5. 配置%HADOOP_HOME%\etc\hadoop\mapred-site.xml；

   <configuration>  
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  
   </configuration>
   

6. 配置%HADOOP_HOME%\etc\hadoop\yarn-site.xml;

   <configuration>  
   <!-- Site specific YARN configuration properties -->  
      <property>  
           <name>yarn.nodemanager.aux-services</name>  
           <value>mapreduce_shuffle</value>  
      </property>  
      <property>  
           <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>    
           <value>org.apache.hadoop.mapred.ShuffleHandler</value>  
      </property>  
   </configuration>
   

7. 配置%HADOOP_HOME%\etc\hadoop\hadoop-env.cmd;

   set JAVA_HOME=D:/WorkTools/Oracle/Java/jdk1.8.0_201
   

8. 初始化 ；
   在CMD中执行：

   hdfs namenode -format
   

9. 启动
   在CMD中执行：

   start-dfs.cmd
   start-yarn.cmd
   

10. 验证
    HDFS : localhost:50070
    YARN : localhost:8088

解决Windows.createDirectoryWithMode0

1. 添加 winutils 下载页面：https://github.com/steveloughran/winutils
2. 建议采用 git clone git@github.com:steveloughran/winutils.git 的方式；
   
   第一次我选择的是2.6.0的，但是没有解决问题，第二次选择3.0.0的。
3. 复制hadoop-3.0.0/bin中的全部文件去替换%HADOOP_HOME%/bin中的文
4. 在要运行的类中添加静态代码块。

   static {
   	try {
      	 	System.load("D:/WorkTools/Hadoop/bin/hadoop.dll");//建议采用绝对地址。
    	} catch (UnsatisfiedLinkError e) {
       	System.err.println("Native code library failed to load.\n" + e);
       	System.exit(1);
   	}
   }
   

   类似于这样：

   public class Demo {
   static {
   
      try {
   
          System.load("D:/WorkTools/Hadoop/bin/hadoop.dll");
   
      } catch (UnsatisfiedLinkError e) {
   
          System.err.println("Native code library failed to load.\n" + e);
   
          System.exit(1);
   
      }
   
   }
   public static void main(String[] args) throws Exception{
   

5. 启动程序验证。

   Counters: 30
   File System Counters
   	FILE: Number of bytes read=858955041
   	FILE: Number of bytes written=27104946
   	FILE: Number of read operations=0
   	FILE: Number of large read operations=0
   	FILE: Number of write operations=0
   Map-Reduce Framework
   	Map input records=300000
   	Map output records=300000
   	Map output bytes=3600000
   	Map output materialized bytes=4200036
   	Input split bytes=954
   	Combine input records=0
   	Combine output records=0
   	Reduce input groups=1
   	Reduce shuffle bytes=4200036
   	Reduce input records=300000
   	Reduce output records=1
   	Spilled Records=600000
   	Shuffled Maps =6
   	Failed Shuffles=0
   	Merged Map outputs=6
   	GC time elapsed (ms)=6
   	Total committed heap usage (bytes)=3909615616
   Shuffle Errors
   	BAD_ID=0
   	CONNECTION=0
   	IO_ERROR=0
   	WRONG_LENGTH=0
   	WRONG_MAP=0
   	WRONG_REDUCE=0
   File Input Format Counters 
   	Bytes Read=173576072
   File Output Format Counters 
   	Bytes Written=19