java.lang.RuntimeException: native-lzo library not available

flume配置lzo压缩将数据存在hdfs上,报错信息如下:

20/05/30 11:42:50 INFO hdfs.BucketWriter: Creating /origin_data/gmall/log/topic_event/2020-05-29/logevent-.1590810087777.lzo.tmp
20/05/30 11:42:50 ERROR hdfs.HDFSEventSink: process failed
java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:97)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
	at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/05/30 11:42:50 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:97)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
	at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

看了一眼hdfs,上面有很多大小为0的以.lzo.tmp结尾的小文件,于是停掉服务开始找错

在经过了格式化namenode、从头到尾检查及同步配置文件、重启服务后还是没有解决,于是格式化namenode、删除所有组件的日志文件、安装lzo(yum install lzo,安装完后突然想起没指定版本,想卸掉重装,吃个饭的时间忘了。。。我估计不是因为这个)、重启服务还是老样子,然后停掉flume服务,再次查看配置文件对比官网查博客,不知道哪里出问题了,重启flume,它就好了。。。折腾了半天,也不知道是怎么死的。。。

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值