Unable to load native-hadoop

背景

部署好Hadoop集群后，发现native-hadoop不可用。

root@test-hdfs-client-7956fbd4f9-gxsf4:/# hdfs dfs -ls /
2019-07-10 12:36:20,232 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - root supergroup          0 2019-07-09 08:02 /user

调查

• 为了获取更多信息，开启debug日志。在$HADOOP_CONF_DIR/log4j.properties中添加如下行：

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG

• 继续执行hdfs命令

root@test-hdfs-client-7956fbd4f9-gxsf4:/# hdfs dfs -ls /
2019-07-10 12:43:01,887 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2019-07-10 12:43:01,888 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2019-07-10 12:43:01,888 DEBUG util.NativeCodeLoader: java.library.path=/opt/hadoop-3.0.0-cdh6.2.0/lib/native
2019-07-10 12:43:01,888 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - root supergroup          0 2019-07-09 08:02 /user

• 确定java.library.path目录是正确的，再看它下面的文件

root@test-hdfs-client-7956fbd4f9-gxsf4:/# ls -l /opt/hadoop-3.0.0-cdh6.2.0/lib/native
total 9300
drwxr-xr-x 2 501 root    4096 Jun 27 09:25 examples
-rwxr-xr-x 1 501 root  167436 Jun 27 09:09 libhadoop.1.0.0.dylib
-rw-r--r-- 1 501 root  705560 Jun 27 09:09 libhadoop.a
-rwxr-xr-x 1 501 root  167436 Jun 27 09:09 libhadoop.dylib
-rw-r--r-- 1 501 root  907408 Jun 27 09:25 libhadooppipes.a
-rw-r--r-- 1 501 root  318824 Jun 27 09:25 libhadooputils.a
-rwxr-xr-x 1 501 root   78144 Jun 27 09:13 libhdfs.0.0.0.dylib
-rw-r--r-- 1 501 root  277632 Jun 27 09:13 libhdfs.a
-rwxr-xr-x 1 501 root   78144 Jun 27 09:13 libhdfs.dylib
-rwxr-xr-x 1 501 root  447940 Jun 29 06:23 libnativetask.1.0.0.dylib
-rw-r--r-- 1 501 root 5900080 Jun 29 06:23 libnativetask.a
-rwxr-xr-x 1 501 root  447940 Jun 29 06:23 libnativetask.dylib

• linux系统下需要libhadoop.so文件 参考

解决

为什么我的native下是libhadoop.dylib，我猜因为我是在mac系统下编译的。解决办法就是在linux系统下重新编译！！！