logstash 使用webhdfs 上传文件出现的问题

记录一次 logstash 6.5.4 使用webhdfs插件向hdfs导入数据错误:

[WARN ][logstash.outputs.webhdfs ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"LogStash::Error", 
:backtrace=>[
"org/logstash/ext/JrubyEventExtLibrary.java:177:in `sprintf'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:194:in `block in flush'", 
"org/jruby/RubyArray.java:2481:in `collect'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:189:in `flush'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'",
"org/jruby/RubyHash.java:1343:in `each'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", 
"/export/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:182:in `receive'", 
"/export/logstash-6.5.4/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'",
"org/jruby/RubyArray.java:1734:in `each'", 
"/export/logstash-6.5.4/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", 
"org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", 
"org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", 
"/export/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:373:in `block in output_batch'", 
"org/jruby/RubyHash.java:1343:in `each'", "/export/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:372:in `output_batch'", 
"/export/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:324:in `worker_loop'", 
"/export/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}

经过分析:
通过:exception=>"LogStash::Error" 可以知道是logstash的原因,因此就关注一下logstash的配置或者操作,又发现每个错都会有这样一段logstash/outputs/webhdfs.rb,所以判断通过是输出webhdfs的配置使用不当的原因,在官网里的推荐配置是这样
在这里插入图片描述

而我没有完全按照该格式

解决方法:

  1. path最终我替换成了/user/logstash/logs/output.txt
  2. 加上dt=

具体的原因,由于无法看到源码,因此暂时未知。。。。

  • 1
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值