Spark最终结果压缩
sparkContext.textFile(inputPath).flatMap(x => {
TestLog.readFromString(x)
}).saveAsTextFile(outputPath, classOf[GzipCodec])
MapReduce最终结果的压缩
FileOutputFormat.setCompressOutput(job, true);//设置压缩
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);//压缩类型
Mapper 中间结果压缩
Configuration conf = new Configuration();
conf.setBoolean("mapred.compress.map.output",true);
conf.setClass("mapred.map.output.compression.codec",GzipCodec.class,CompressionCodec.class);