查看使用linkedIn Camus 把Kafka中的数据导入HDFS中生成的.deflate文件

在使用Camus好不容易把kafka中的数据导入了HDFS,但是直接download后打开,显示的会是乱码。经查询,带.deflate后缀的文件是使用DEFLATE算法压缩过的,所以要查看,只需使用hadoop的命令即可:

hadoop dfs -text /user/kafka-topic.deflate

展开阅读全文

用java读取hdfs的.lzo_deflate文件报错

09-09
linux环境没有问题,hadoop环境、配置也没有问题,并且通过hdoop fs -text 指令能正常打开该压缩文件。但是用java读取就报错了,请大神帮忙看看,谢谢 代码如下: public static void main(String[] args) { String uri = "/daas/****/MBLDPI3G.2016081823_10.1471532401822.lzo_deflate"; Configuration conf = new Configuration(); String path = "/software/servers/hadoop-2.6.3-bin/hadoop-2.6.3/etc/hadoop/"; conf.addResource(new Path(path + "core-site.xml")); conf.addResource(new Path(path + "hdfs-site.xml")); conf.addResource(new Path(path + "mapred-site.xml")); try { CompressionCodecFactory factory = new CompressionCodecFactory(conf); CompressionCodec codec = factory.getCodec(new Path(uri)); if (codec == null) { System.out.println("Codec for " + uri + " not found."); } else { CompressionInputStream in = null; try { in = codec.createInputStream(new java.io.FileInputStream(uri)); byte[] buffer = new byte[100]; int len = in.read(buffer); while (len > 0) { System.out.write(buffer, 0, len); len = in.read(buffer); } } finally { if (in != null) { in.close(); } } } } catch (Exception e) { e.printStackTrace(); } } 报错信息如下: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. java.io.FileNotFoundException: /daas/***/MBLDPI3G.2016081823_10.1471532401822.lzo_deflate (没有那个文件或目录) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at FileDecompressor.main(FileDecompressor.java:53) 加载的jar包: <classpathentry kind="lib" path="lib/commons-cli-1.2.jar"/> <classpathentry kind="lib" path="lib/commons-collections-3.2.2.jar"/> <classpathentry kind="lib" path="lib/commons-configuration-1.6.jar"/> <classpathentry kind="lib" path="lib/commons-lang-2.6.jar"/> <classpathentry kind="lib" path="lib/commons-logging-1.1.3.jar"/> <classpathentry kind="lib" path="lib/guava-18.0.jar"/> <classpathentry kind="lib" path="lib/hadoop-auth-2.6.3.jar"/> <classpathentry kind="lib" path="lib/hadoop-common-2.6.3.jar"/> <classpathentry kind="lib" path="lib/hadoop-hdfs-2.6.3.jar"/> <classpathentry kind="lib" path="lib/htrace-core-3.0.4.jar"/> <classpathentry kind="lib" path="lib/log4j-1.2.17.jar"/> <classpathentry kind="lib" path="lib/protobuf-java-2.5.0.jar"/> <classpathentry kind="lib" path="lib/slf4j-api-1.7.5.jar"/> <classpathentry kind="lib" path="lib/slf4j-log4j12-1.7.5.jar"/> <classpathentry kind="lib" path="lib/hadoop-lzo-0.4.20.jar"/>
©️2020 CSDN 皮肤主题: 大白 设计师: CSDN官方博客 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值