hadoop的snappy压缩出错

已于 2022-08-28 16:21:58 修改
阅读量1.3k 收藏 1
点赞数
文章标签： spark scala hadoop
于 2020-10-30 17:38:39 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/SJH2725970073/article/details/109389870
版权

说明你的本地hadoop没有安装snappy压缩。
可以使用自己编译snappy，或者使用已安装好snappy压缩的
hadoop替换自己原来的hadoop。hadoop3.x带snappy(可用于windows本地开发)

下面写一下怎么用，这个压缩包。
1、 解压这个文件
在这里插入图片描述
可以看到 解压之后的文件是有winutils.exe 和 snappy.dll的，我没有忽悠吧
2、把HADOOP_HOME的环境变量配置一下就好
3、 创建工程（略）

我给大家测试一下：

这个一个snappy压缩的文件
我使用以下代码去读取

import org.apache.spark.sql.SparkSession

object Test {
  def main(args: Array[String]): Unit = {
    System.setProperty("HADOOP_USER_NAME", "hdfs")
    System.setProperty("HADOOP_HOME", "D:\\\\java-dev\\hadoop-3.0.0")
    val spark = SparkSession.builder().master("local[*]").getOrCreate()
    spark.read.text("file:///D:\\IdeaProjects\\Spark\\sparkdemo\\input file\\word\\part-00000.snappy")
      .show()
  }
}

在这里插入图片描述

java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
	at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
	at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:175)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1915)
	at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
	at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
	at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:252)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:251)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/10/30 17:32:47 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
	at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
	at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:175)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1915)
	at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
	at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
	at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:252)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:251)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/10/30 17:32:47 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, ANY, 6161 bytes)
20/10/30 17:32:47 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
20/10/30 17:32:47 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, ANY, 6204 bytes)
20/10/30 17:32:47 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
20/10/30 17:32:47 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
	at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
	at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:175)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1915)
	at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
	at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
	at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:252)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:251)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
飞天小虎
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
winrar 解压 hadoop-2.7.3-src.tar.gz 失败及解决办法
陆树勤的博客
10-31 5345
问题#1： GetTaskAttemptCompletionEventsRequestPBImpl.java 解压失败 ------------------------------------------------------------- !   F:\解包测试\hadoop-2.7.3-src.rar: 无法创建 hadoop-2.7.3-src\hadoop-mapreduce-pro...
Hadoop2.6.0源码编译支持Snappy压缩
浮生、佐以酒
01-12 804
环境准备 系统:Centos6.5 64位 JDK1.7 Maven：3.6.0 Hadoop源码：Hadoop2.6.0-cdh5.7.0.src.tar.gz Protocol Buffer 2.5.0 Jar依赖包 安装Protocol Buffer 2.5.0 root权限安装依赖包 $ yum install -y gcc gcc-c++ make cmake 指定安装目录...
Hadoop安装文件解压报错：无法创建符号链接。。。
06-12 2185
Hadoop安装文件解压报错：无法创建符号链接，无权限。。。
压缩归档库-Snappy介绍
最新发布
希望能帮助大家，希望大家多多支持，你们的支持是我前进的动力。
05-09 813
Snappy 是一个 C++ 编写的压缩和解压缩库，由 Google 开发。它专为速度而设计，而不是最大压缩率或与其他压缩库的兼容性。Snappy 通常用于需要快速压缩和解压缩的场景。快速：压缩速度达到250 MB/秒及以上。稳定：Snappy在谷歌的生产环境中压缩和解压缩了数PB的数据。Snappy比特流格式是稳定的，不会在不同版本之间发生变化。鲁棒性：Snappy解压缩器的设计不会在面对损坏或恶意输入时崩溃。免费开源软件：Snappy是在BSD类型的许可证下获得许可的。
hadoopsnappy解压_Hadoop安装配置snappy压缩
weixin_34902131的博客
01-17 681
本文的主要详细介绍hadoop2.x中snappy压缩算法安装配置的详细步骤。[一]、 实验环境CentOS 6.3 64位Hadoop 2.6.0JDK1.7.0_75[二]、 snappy编译安装2.1、下载源码2.2、编译安装解压tar -zxvf snappy-1.1.1.tar.gz,然后以root用户 执行标准的三步进行编译安装：./configuremakemake ins...
最笨的方法解决 使用Snappy 压缩方式报错“java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path”...
weixin_30374009的博客
08-31 318
之前写过一篇这个文章：http://blog.csdn.net/stark_summer/article/details/47361603，那个时候 linux环境 spark 使用snappy方式压缩任然不好用，而今天我同事hive on hadoop 使用snappy压缩方式也报这个错，此刻的我，感觉这个问题 一定要解决 我想了想，只能使用最笨的方式先解决这个问题了，将libsnap...
hadoop3.x带snappy(可用于windows本地开发)
10-26
编译过的hadoop3.0版本，安装了snappy压缩。
Hadoop 安装snappy（编译源码）
盖世英雄来了
09-29 790
首先，由于下载的hadoop一般都是编译好的，不支持snappy压缩，因此，我们需要下载未编译的hadoop进行编译，然后自己设置支持的压缩格式。以下开始详细配置： 安装之前 hadoop checknative -a 1.准备 ###安装其中部分基础软件的命令： yum -y install gcc gcc-c++ libtool cmake maven zlib-devel 下载...
windows下的hadoop环境以及snappy的问题
Luckin的博客
05-20 693
关于windows下的hadoop环境配置请参照以下链接 https://blog.csdn.net/qq_35139965/article/details/106744410?utm_medium=distribute.pc_relevant.none-task-blog-baidujs_baidulandingword-4&spm=1001.2101.3001.4242 关于snappy的native library的问题，IDEA无法直接连接远程hive表压缩数据的问题 请下载添加sna
hadoop-snappy-0.0.1-SNAPSHOT.tar.gz
11-09
hadoop hbase支持snappy压缩jar包 解压后放到hbaselib目录重启
hadoop2.7.4-snappy
11-14
编译hadoop2.7.4源码，使之支持snappy压缩存储。绝对正确
hadoop 2.6 native snappy
06-26
编译了2个多小时编译出来的，分享出来给大家，hadoop2.6 native本地库文件，支持snappy压缩(注：网上下载的大部分不支持snappy)
hadoop-snappy的jar包
03-15
hadoop中使用snappy算法，需要手工编译hadoop-snappy的包，但是这个包没有提供下载，需要花很大力气进行编译，现在把这个包提供出来
Hadoop压缩-SNAPPY算法
yydcj的专栏
04-10 1905
本篇文章做了小部分更改，仅介绍了Snappy，去掉了安装过程，不过不必叹气，更加详细的Hadoop Snappy及HBase Snappy的安装步骤已经另起了一篇文章专门来介绍：Hadoop HBase 配置 安装 Snappy 终极教程 通过这篇文章，相信你一定会几乎毫无困难的成功安装Snappy。 Compression就是在用CPU换IO吞吐量/磁盘空间，如果没有什么特殊原因推荐针
hadoop的压缩方式snappy
RZH_long的博客
11-23 2031
文件压缩有两大好处，节约磁盘空间，加速数据在网络和磁盘上的传输 在重新编译过hadoop的c源码后才可以用谷歌的snappy压缩。 我们可以使用bin/hadoop checknative 来查看我们编译之后的hadoop支持的各种压缩，如果出现openssl为false，那么就在线安装一下依赖包 //查看压缩支持情况 bin/hadoop checknative //在线安装 yum install openssl-devel 这种压缩对应的java类 压缩格式 ...
hadoop，hbase添加snappy压缩算法
qq_43680345的博客
11-13 453
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。 本文链接： https://mp.csdn.net/mdeditor/103046844 参考博文： http://blog.cheyo.net/197.html https://blog.csdn.net/bbaiggey/article/details/53583923 https://g...
测试hadoop安装成功与失败
大数据训练营
01-04 8177
1.首先是jps,看看java进程是否都在 2.查看hdfs和yarn的web地址,看看能否出现 3.hdfs dfs -ls / ,看看目录是否列的出来 4../hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 5 10 跑一个mr例子试试 这样就差不多了
Hadoop安装snappy（编译源码）
蔡政洁的博客
10-25 1175
目录了解Hive的数据压缩Hadoop安装snappy（编译源码）第一步：配置阿里云yum源第二步：配置/etc/resolv.conf第三步：下载Hadoop源码并解压第四步：配置JDK第五步：安装配置maven第六步：protobuf-2.5.0安装第七步：snappy安装第八步：安装CMake第九步：hadoop编译第十步：hadoop配置修改（1）修改core-site.xml（2）修改mapred-site.xml测试疑惑详解 了解Hive的数据压缩 hive 的数据压缩 == MR的数据压缩
this version of libhadoop was built without snappy support. 解决方案
minghai
08-20 906
线上遇到该问题，hadoop checknative 显示是支持 Snappy 压缩的 [yarn@Centos hadoop]$ bin/hadoop checknative -a 21/08/20 17:52:39 INFO conf.Configuration: Current hadoop.security.authentication = simple 21/08/20 17:52:39 INFO bzip2.Bzip2Factory: Successfully loaded & init
java的spark2屏蔽 hadoop snappy压缩
06-10
在Java的Spark2中，您可以通过设置以下属性来屏蔽Hadoop Snappy压缩： ``` spark.hadoop.io.compression.codec.snappy.use.native=false ``` 您可以将其添加到SparkConf中，如下所示： ```java SparkConf conf = ...
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值