Hadoop | SequenceFile doesn't work with GzipCodec without native-hadoop code 异常解决-CSDN博客

最近莫名其妙的碰到了这个Hadoop异常:SequenceFile doesn't work with GzipCodec without native-hadoop code!这一般跟环境配置有关,以前也碰到过,但好久没碰了,如何解决没啥印象.就从零开始解决吧.
第一眼看这个异常感觉跟我lzo没配置好有关,因为我使用的是Lzo压缩.不是什么Gzip.于是调试了下lzo压缩,可以通过. 于是只能看源代码了.原来是在没有reducer,hadoop建立OutputFormat的时候,需要构建Writer,所触发.触发代码如下:
if ((codec instanceof GzipCodec) &&
!NativeCodeLoader.isNativeCodeLoaded() &&
!ZlibFactory.isNativeZlibLoaded(conf)) {
throw new IllegalArgumentException("SequenceFile doesn't work with " +
"GzipCodec without native-hadoop code!");
}
所以有两种情况会触发这个异常,一种是NativeCodeLoader.isNativeCodeLoaded()=false,还有一种是ZlibFactory.isNativeZlibLoaded(conf)=false
1.先介绍NativeCodeLoader.isNativeCodeLoaded()
NativeCodeLoader.isNativeCodeLoaded()是加载本地Hadoop库.代码如下:
static {
// Try to load native hadoop library and set fallback flag appropriately
LOG.debug("Trying to load the custom-built native-hadoop library...");
try {

System.loadLibrary("hadoop");

LOG.info("Loaded the native-hadoop library");
nativeCodeLoaded = true;
} catch (Throwable t) {
// Ignore failure to load
LOG.debug("Failed to load native-hadoop with error: " + t);
LOG.debug("java.library.path=" + System.getProperty("java.library.path"));
}
if (!nativeCodeLoaded) {
LOG.warn("Unable to load native-hadoop library for your platform... " +
"using builtin-java classes where applicable");
}
}
我是这里 System.loadLibrary("hadoop");触发了异常,但日志是DEBUG.所以没有看到.但如果你设置了Hadoop日志等级为DEBUG
的话,会看到如下异常:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
java.library.path=/usr/local/lib
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
所以需要将hadoop库移到你的java.library.path下,我这里是/usr/local/lib.你可以通过System.getProperty("java.library.path")获取.
有人会问hadoop库在哪里,一般存在与你的HADOOP_HOME/lib/native,然后根据你操作系统的位数选择是Linux-amd64-64还是Linux-i386-32.然后将对应该目录的文件下的所有文件拷贝到你的System.getProperty("java.library.path")目录下即可.
2.再介绍ZlibFactory.isNativeZlibLoaded(conf):
主要是加载压缩和解压缩器,代码如下:
static {
if (NativeCodeLoader.isNativeCodeLoaded()) {
nativeZlibLoaded = ZlibCompressor.isNativeZlibLoaded() &&
ZlibDecompressor.isNativeZlibLoaded();
if (nativeZlibLoaded) {
LOG.info("Successfully loaded & initialized native-zlib library");
} else {
LOG.warn("Failed to load/initialize native-zlib library");
}
}