Hadoop-3.3.4集群部分lib缺失问题

1.问题描述

(base) [hadoop@hadoop1 native]$ hadoop checknative
2023-12-25 14:20:21,615 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2023-12-25 14:20:21,618 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2023-12-25 14:20:21,623 WARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
2023-12-25 14:20:21,623 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2023-12-25 14:20:21,658 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /data/module/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
PMDK:    false The native code was built without PMDK support.

2.原因分析

  • Failed to load libisal.so.2
    缺少libisal.so.2,需要安装
  • Cannot load libcrypto.so
    必要软件包

3.问题解决

3.1 解决Failed to load libisal.so.2

Intel ISA-L代表英特尔智能存储加速库。ISA-L是针对存储应用程序而优化的低级功能的开源集合。它包括针对Intel AVX和AVX2指令集优化的快速块Reed-Solomon类型纠删码。HDFS纠删码可以利用ISA-L加速编码和解码计算。ISA-L支持大多数主要操作系统,包括Linux和Windows。默认情况下不启用ISA-L。

  1. 安装yasm和nasm
sudo yum install -y yasm
sudo yum install -y nasm

注意:isa-l-2.30.0对nasm和yasm有版本要求,低版本在安装时会报错。

  1. 编译安装isa-l-2.30.0
tar -zxvf isa-l-2.30.0.tar.gz
cd isa-l-2.30.0
sudo ./autogen.sh
sudo  ./configure --prefix=/usr --libdir=/usr/lib64
sudo  make
sudo  make install
 
sudo make -f Makefile.unx

各部操作无报错表示编译安装成功。

  1. 检查libisal.so*是否成功

ll /lib64/libisal.so*
 
############如果有,则跳过##############

############如果没有有,则复制##############
cp bin/libisal.so bin/libisal.so.2 /lib64
  1. 复制编译好的libisal.so.2/libisal.so至其他机器的/lib64目录
sudo xsync libisal.so.2 libisal.so
  1. 在Hadoop上运行检查命令
hadoop checknative

3.2 解决Cannot load libcrypto.so

参考hadoop官网说明https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

 sudo  yum -y install  lzo-devel  zlib-devel  gcc gcc-c++ autoconf automake libtool openssl-devel fuse-devel cmake

4.测试是否完成

[hadoop@hadoop2 bin]$ hadoop checknative
2023-12-25 16:30:01,759 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2023-12-25 16:30:01,762 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2023-12-25 16:30:01,793 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /data/module/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2
PMDK:    false The native code was built without PMDK support.
  • 8
    点赞
  • 8
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值