排查logstash2.4升级到5.0版本后kafka不兼容问题

鎺掓煡鍗囩骇鍒扮増鏈悗kafka涓嶅吋瀹归棶棰?/p>

鍙傝€冩枃妗ｏ細

/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-output-kafka-/lib/logstash/outputs/

缂樼敱锛?/p>

涔嬪墠瀵笶LKB鐜浠庣増鏈崌绾у埌鏈€鏂扮殑绋冲畾鐗堟湰锛屼富瑕佸崌绾ф楠ゅ彲浠ュ弬鑰?3002256/1870205锛屽悗鏉ュ彂鐜発afka闆嗙兢杩愯鎶ラ敊锛岀幇鍦ㄦ妸鎺掓煡杩囩▼璁板綍濡備笅锛屼粎渚涘弬鑰?/p>

涔嬪墠鐜锛?/p>

logstash-input-kafka-

logstash-output-kafka-

kafka_-

鍗囩骇鍚庣幆澧冿細

logstash-input-kafka-

logstash-output-kafka-

鎶ラ敊淇℃伅锛?/p>

[2016-11-16T14:35:44,739][ERROR][    ] Unknown setting 'zk_connect' for kafka
[2016-11-16T14:35:44,741][ERROR][    ] Unknown setting 'topic_id' for kafka
[2016-11-16T14:35:44,741][ERROR][    ] Unknown setting 'reset_beginning' for kafka

瀹炴柦姝ラ锛?/p>

1锛屾牴鎹敊璇煡鐪嬬▼搴忓摢閲屾姤閿?/p>

grep  "Unknown setting" /usr/share/logstash/ -R
/usr/share/logstash/logstash-core/lib/logstash/config/:          ("Unknown setting '#{name}' for #{@plugin_name}")

2锛屾煡鐪嬬▼搴忕浉鍏充唬鐮侊紝鍙戠幇闇€瑕佹煡鐪媝lugins鐨刢onfig瀹氫箟鏂囦欢绛?/p>

    def validate_check_invalid_parameter_names(params)
      invalid_params = 
      # Filter out parameters that match regexp keys.
      # These are defined in plugins like this:
      #   config /foo.*/ => ...
      @_key do |config_key|
        if config__a?(Regexp)
          invalid_! { |k| k =~ config_key }
        elsif config__a?(String)
          invalid_! { |k| k == config_key }
        end
      end
      if invalid_ > 0
        invalid_ do |name|
          ("Unknown setting '#{name}' for #{@plugin_name}")
        end
        return false
      end # if invalid_ > 0
      return true
    end # def validate_check_invalid_parameter_names

3锛岃繘鍏ユ彃浠舵€荤洰褰曟煡鐪嬪叿浣撲俊鎭?/p>

cd  /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-

鍙戠幇閲嶇偣鏌ョ湅濡備笅鏂囦欢

grep config ./* -R |awk '{print $1}' |uniq 
./:
./:See
./lib/logstash/inputs/:#
./lib/logstash/inputs/:
./:-
Binary

1锛夐鍏堢湅锛屽氨鏈夊彂鐜發ogstash-input-寮€濮嬪氨涓嶅湪鍚戝悗鍏煎锛屼笖鍓旈櫎浜唈ruby-kafka锛屾敞鎰忚繖閲屾湁涓潙2锛変細璁插埌锛岀増鏈寮€濮嬫敮鎸乲afka 锛屽張璇村紑濮?/p>

鏀寔鍒囦笉鍚戝悗鍏煎锛岃繖鐮村潖鎬ф洿鏂颁篃鏄浜嗐€傜湅鏉ラ棶棰樻壘鍒颁簡鎴戠殑kafka鐗堟湰鏄痥afka_-锛宬afka鐗堟湰涓嶅吋瀹瑰鑷寸殑銆? 

閮ㄥ垎鏂囨。濡備笅锛?/p>

## 
  - Update to Kafka version  for bug fixes
## 
  - Support for Kafka  which is not backward compatible with  broker.
## 
  - Republish all the gems under jruby.
  - Update the plugin to the version  of the plugin api, this change is required for Logstash  compatibility. See /elastic/logstash/issues/5141
  - Support for Kafka  for LS 
## 
 - Refactor to use new Java based consumer, bypassing jruby-kafka
 - Breaking: Change configuration to match Kafka's configuration. This version is not backward compatible

2锛変箣鍓嶆垜鐪嬫枃妗ｆ椂,鐪嬮厤缃娉曢兘姝ｇ‘锛岃繕浠ヤ负鏄嵈灏戜緷璧栧叧绯籮ruby-kafka library鍛紝杩欎釜鍐嶆槸鍦ㄧ敤鐨?鍙﹀瀵规瘮鍙戠幇5鐗堟湰灏戜簡涓嶅皯鎻掍欢銆傚彟澶?/p>

kafka鐗堟湰鍐欑殑鏄紝鐪嬫潵杩欎釜娌℃湁鍙婃椂鏇存柊锛堜笌鍚庨潰鏂囦欢涓嶄竴鑷达級锛岃皝瑕佹槸鐪嬪埌浜嗛夯鐑﹀強鏃舵洿鏂板晩锛岃櫧鏄皬闂浣嗘槸涔熷彲鑳借瀵兼垜绛夊眮姘戙€傚綋鐒朵篃鏈夊彲鑳芥槸鎴戞病

鏈夊叏闈㈢湅鏂囨。瀵艰嚧鐨勩€?/p>

鏂囨。缁撳熬濡備笅锛?/p>

Dependencies
====================
* Apache Kafka version 
* jruby-kafka library

3锛夊紑濮嬬湅鏂囨。锛岀壒鎰忕湅浜嗕笅kafka鐨勫吋瀹规€?鐪嬫潵logstas-input-鍜宭ogstash-output-鍙兘鐢ㄤ簡銆傚鏋滀綘鎯崇敤杩樻兂鐢?浣犵殑

logstash-input-kafka鍜宭ogstash-output-kafka鍙兘闄嶇骇鐗堟湰鍒颁簡锛岃繖閲岄兘璇翠粬鏄腑闂磋繃娓＄増鏈簡锛屾墍浠ヨ繕鏄殢澶ф祦鍚с€?/p>

## Kafka Compatibility
Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers 
since brokers target backwards compatibility. The  broker will work with both the  consumer and  consumer APIs but not the other way around.
| Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? |
|:---------------:|:------------------:|:--------------:|:---------------:|:------|
|            |  -    | <  | < | Legacy,  is still popular |
|            |  -    |    |   | Intermediate release before  that works with old Ruby Event API `[]`  |
|           | ,            |    |   | Intermediate release before  with new get/set API |
|          | ,            |    |   | Track latest Kafka release. Not compatible with  broker |

4锛夌幇鍦ㄧ湅鏉ュ彧鑳藉崌绾afka鐗堟湰浜嗐€傛渶鍚庢垜鐪嬩簡涓媕ar-dependencies鍙戠幇浜唊afka-clients-

ls /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/vendor/jar-dependencies/runtime-jars/
kafka-clients-  log4j-  lz4-  slf4j-api-  slf4j-log4j12-  snappy-java-

5锛夎繕鏈変竴涓枃浠舵病鏈夌湅锛屾€€鐫€濂藉蹇冩垜鐪嬩簡涓€鐪硷紝鍙戠幇涔嬪墠閮界櫧璐瑰姏姘斾簡锛岃繖閲屾墠鏄渶鏈夊弬鑰冧环鍊肩殑鐨勪富鍙傝€冩枃妗ｅ晩锛屾槸鎹峰緞鍟婏紝闅愯棌鐨勫娣辩殑锛屽樊鐐归敊杩囦簡锛屾睏锛?/p>

/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/

閮ㄥ垎鏂囨。濡備笅锛?/p>

# This input will read events from a Kafka topic. It uses the the newly designed
#  version of consumer API provided by Kafka to read messages from the broker.
#
# Here's a compatibility matrix that shows the Kafka client versions that are compatible with each combination
# of Logstash and the Kafka input plugin: 
# 
# [options="header"]
# |==========================================================
# |Kafka Client Version |Logstash Version |Plugin Version |Security Features |Why?
# |       | -    |< | |Legacy,  is still popular 
# |       | -    |  |Basic Auth, SSL |Works with the old Ruby Event API (`event['product']['price'] = 10`)  
# |       | -    |  |Basic Auth, SSL |Works with the new getter/setter APIs (`('[product][price]', 10)`)
# |      | -    |  |Basic Auth, SSL |Not compatible with the  broker 
# |==========================================================
# 
# NOTE: We recommended that you use matching Kafka client and broker versions. During upgrades, you should
# upgrade brokers before clients because brokers target backwards compatibility. For example, the  broker
# is compatible with both the  consumer and  consumer APIs, but not the other way around.

6锛夊崌绾afka_-涓簁afka_- 锛堟垜鐪媖afka-clients-锛屾墍浠ユ病鏈夌敤鏈€鏂扮殑kafka_-锛?/p>

澶ф姝ラ

鍏抽棴鑰乲afka

/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/

澶囦唤鑰侀厤缃枃浠?/p>

鍜?/p>

鍒犻櫎kafka

rm -rf /usr/local/kafka/

rm -rf /data/kafkalogs/*

瀹夎閰嶇疆鏂発afka

wget /apache/kafka//kafka_-

tar zxvf kafka_- -C /usr/local/

ln -s /usr/local/kafka_- /usr/local/kafka

diff浜嗕笅鍜屽彉鍔ㄤ笉澶у彲浠ョ洿鎺ヤ娇鐢?/p>

鍚姩鏂発afka

/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/ &

7)娉ㄦ剰鍑犱釜鍏抽敭閰嶇疆闇€瑕佷慨鏀?/p>

config :bootstrap_servers, :validate => :string, :default => "localhost:9092"

config :group_id, :validate => :string, :default => "logstash"

config :topics, :validate => :array, :default => ["logstash"]

config :consumer_threads, :validate => :number, :default => 1

闄や簡涓婇潰鐨勫嚑涓叧閿厤缃锛宬afka鐨則opic鍒嗙墖淇℃伅闇€瑕侀噸鏂癱reate涓€浠斤紝鍚﹀垯KafkaMonitor鐩戞帶涓嶅嚭Active Topic Consumer鍥惧舰锛屼絾瀹為檯鏄湪宸ヤ綔涓€?/p>