背景
部署好Hadoop集群后,发现native-hadoop不可用。
root@test-hdfs-client-7956fbd4f9-gxsf4:/# hdfs dfs -ls /
2019-07-10 12:36:20,232 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - root supergroup 0 2019-07-09 08:02 /user
调查
- 为了获取更多信息,开启debug日志。在$HADOOP_CONF_DIR/log4j.properties中添加如下行:
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG
- 继续执行hdfs命令
root@test-hdfs-client-7956fbd4f9-gxsf4:/# hdfs dfs -ls /
2019-07-10 12:43:01,887 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2019-07-10 12:43:01,888 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2019-07-10 12:43:01,888 DEBUG util.NativeCodeLoader: java.library.path=/opt/hadoop-3.0.0-cdh6.2.0/lib/native
2019-07-10 12:43:01,888 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - root supergroup 0 2019-07-09 08:02 /user
- 确定java.library.path目录是正确的,再看它下面的文件
root@test-hdfs-client-7956fbd4f9-gxsf4:/# ls -l /opt/hadoop-3.0.0-cdh6.2.0/lib/native
total 9300
drwxr-xr-x 2 501 root 4096 Jun 27 09:25 examples
-rwxr-xr-x 1 501 root 167436 Jun 27 09:09 libhadoop.1.0.0.dylib
-rw-r--r-- 1 501 root 705560 Jun 27 09:09 libhadoop.a
-rwxr-xr-x 1 501 root 167436 Jun 27 09:09 libhadoop.dylib
-rw-r--r-- 1 501 root 907408 Jun 27 09:25 libhadooppipes.a
-rw-r--r-- 1 501 root 318824 Jun 27 09:25 libhadooputils.a
-rwxr-xr-x 1 501 root 78144 Jun 27 09:13 libhdfs.0.0.0.dylib
-rw-r--r-- 1 501 root 277632 Jun 27 09:13 libhdfs.a
-rwxr-xr-x 1 501 root 78144 Jun 27 09:13 libhdfs.dylib
-rwxr-xr-x 1 501 root 447940 Jun 29 06:23 libnativetask.1.0.0.dylib
-rw-r--r-- 1 501 root 5900080 Jun 29 06:23 libnativetask.a
-rwxr-xr-x 1 501 root 447940 Jun 29 06:23 libnativetask.dylib
- linux系统下需要libhadoop.so文件 参考
解决
为什么我的native下是libhadoop.dylib
,我猜因为我是在mac系统下编译的。解决办法就是在linux系统下重新编译!!!