鎺掓煡鍗囩骇鍒扮増鏈悗kafka涓嶅吋瀹归棶棰?/p>
鍙傝€冩枃妗o細
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-output-kafka-/lib/logstash/outputs/
缂樼敱锛?/p>
涔嬪墠瀵笶LKB鐜浠庣増鏈崌绾у埌鏈€鏂扮殑绋冲畾鐗堟湰锛屼富瑕佸崌绾ф楠ゅ彲浠ュ弬鑰?3002256/1870205锛屽悗鏉ュ彂鐜発afka闆嗙兢杩愯鎶ラ敊锛岀幇鍦ㄦ妸鎺掓煡杩囩▼璁板綍濡備笅锛屼粎渚涘弬鑰?/p>
涔嬪墠鐜锛?/p>
logstash-input-kafka-
logstash-output-kafka-
kafka_-
鍗囩骇鍚庣幆澧冿細
logstash-input-kafka-
logstash-output-kafka-
鎶ラ敊淇℃伅锛?/p>
[2016-11-16T14:35:44,739][ERROR][ ] Unknown setting 'zk_connect' for kafka [2016-11-16T14:35:44,741][ERROR][ ] Unknown setting 'topic_id' for kafka [2016-11-16T14:35:44,741][ERROR][ ] Unknown setting 'reset_beginning' for kafka
瀹炴柦姝ラ锛?/p>
1锛屾牴鎹敊璇煡鐪嬬▼搴忓摢閲屾姤閿?/p>
grep "Unknown setting" /usr/share/logstash/ -R /usr/share/logstash/logstash-core/lib/logstash/config/: ("Unknown setting '#{name}' for #{@plugin_name}")
2锛屾煡鐪嬬▼搴忕浉鍏充唬鐮侊紝鍙戠幇闇€瑕佹煡鐪媝lugins鐨刢onfig瀹氫箟鏂囦欢绛?/p>
def validate_check_invalid_parameter_names(params) invalid_params = # Filter out parameters that match regexp keys. # These are defined in plugins like this: # config /foo.*/ => ... @_key do |config_key| if config__a?(Regexp) invalid_! { |k| k =~ config_key } elsif config__a?(String) invalid_! { |k| k == config_key } end end if invalid_ > 0 invalid_ do |name| ("Unknown setting '#{name}' for #{@plugin_name}") end return false end # if invalid_ > 0 return true end # def validate_check_invalid_parameter_names
3锛岃繘鍏ユ彃浠舵€荤洰褰曟煡鐪嬪叿浣撲俊鎭?/p>
cd /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-
鍙戠幇閲嶇偣鏌ョ湅濡備笅鏂囦欢
grep config ./* -R |awk '{print $1}' |uniq ./: ./:See ./lib/logstash/inputs/:# ./lib/logstash/inputs/: ./:- Binary
1锛夐鍏堢湅锛屽氨鏈夊彂鐜發ogstash-input-寮€濮嬪氨涓嶅湪鍚戝悗鍏煎锛屼笖鍓旈櫎浜唈ruby-kafka锛屾敞鎰忚繖閲屾湁涓潙2锛変細璁插埌锛岀増鏈寮€濮嬫敮鎸乲afka 锛屽張璇村紑濮?/p>
鏀寔鍒囦笉鍚戝悗鍏煎锛岃繖鐮村潖鎬ф洿鏂颁篃鏄浜嗐€傜湅鏉ラ棶棰樻壘鍒颁簡鎴戠殑kafka鐗堟湰鏄痥afka_-锛宬afka鐗堟湰涓嶅吋瀹瑰鑷寸殑銆?
閮ㄥ垎鏂囨。濡備笅锛?/p>
## - Update to Kafka version for bug fixes ## - Support for Kafka which is not backward compatible with broker. ## - Republish all the gems under jruby. - Update the plugin to the version of the plugin api, this change is required for Logstash compatibility. See /elastic/logstash/issues/5141 - Support for Kafka for LS ## - Refactor to use new Java based consumer, bypassing jruby-kafka - Breaking: Change configuration to match Kafka's configuration. This version is not backward compatible
2锛変箣鍓嶆垜鐪嬫枃妗f椂,鐪嬮厤缃娉曢兘姝g‘锛岃繕浠ヤ负鏄嵈灏戜緷璧栧叧绯籮ruby-kafka library鍛紝杩欎釜鍐嶆槸鍦ㄧ敤鐨?鍙﹀瀵规瘮鍙戠幇5鐗堟湰灏戜簡涓嶅皯鎻掍欢銆傚彟澶?/p>
kafka鐗堟湰鍐欑殑鏄紝鐪嬫潵杩欎釜娌℃湁鍙婃椂鏇存柊锛堜笌鍚庨潰鏂囦欢涓嶄竴鑷达級锛岃皝瑕佹槸鐪嬪埌浜嗛夯鐑﹀強鏃舵洿鏂板晩锛岃櫧鏄皬闂浣嗘槸涔熷彲鑳借瀵兼垜绛夊眮姘戙€傚綋鐒朵篃鏈夊彲鑳芥槸鎴戞病
鏈夊叏闈㈢湅鏂囨。瀵艰嚧鐨勩€?/p>
鏂囨。缁撳熬濡備笅锛?/p>
Dependencies ==================== * Apache Kafka version * jruby-kafka library
3锛夊紑濮嬬湅鏂囨。锛岀壒鎰忕湅浜嗕笅kafka鐨勫吋瀹规€?鐪嬫潵logstas-input-鍜宭ogstash-output-鍙兘鐢ㄤ簡銆傚鏋滀綘鎯崇敤杩樻兂鐢?浣犵殑
logstash-input-kafka鍜宭ogstash-output-kafka鍙兘闄嶇骇鐗堟湰鍒颁簡锛岃繖閲岄兘璇翠粬鏄腑闂磋繃娓$増鏈簡锛屾墍浠ヨ繕鏄殢澶ф祦鍚с€?/p>
## Kafka Compatibility Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers since brokers target backwards compatibility. The broker will work with both the consumer and consumer APIs but not the other way around. | Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? | |:---------------:|:------------------:|:--------------:|:---------------:|:------| | | - | < | < | Legacy, is still popular | | | - | | | Intermediate release before that works with old Ruby Event API `[]` | | | , | | | Intermediate release before with new get/set API | | | , | | | Track latest Kafka release. Not compatible with broker |
4锛夌幇鍦ㄧ湅鏉ュ彧鑳藉崌绾afka鐗堟湰浜嗐€傛渶鍚庢垜鐪嬩簡涓媕ar-dependencies鍙戠幇浜唊afka-clients-
ls /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/vendor/jar-dependencies/runtime-jars/ kafka-clients- log4j- lz4- slf4j-api- slf4j-log4j12- snappy-java-
5锛夎繕鏈変竴涓枃浠舵病鏈夌湅锛屾€€鐫€濂藉蹇冩垜鐪嬩簡涓€鐪硷紝鍙戠幇涔嬪墠閮界櫧璐瑰姏姘斾簡锛岃繖閲屾墠鏄渶鏈夊弬鑰冧环鍊肩殑鐨勪富鍙傝€冩枃妗e晩锛屾槸鎹峰緞鍟婏紝闅愯棌鐨勫娣辩殑锛屽樊鐐归敊杩囦簡锛屾睏锛?/p>
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/
閮ㄥ垎鏂囨。濡備笅锛?/p>
# This input will read events from a Kafka topic. It uses the the newly designed # version of consumer API provided by Kafka to read messages from the broker. # # Here's a compatibility matrix that shows the Kafka client versions that are compatible with each combination # of Logstash and the Kafka input plugin: # # [options="header"] # |========================================================== # |Kafka Client Version |Logstash Version |Plugin Version |Security Features |Why? # | | - |< | |Legacy, is still popular # | | - | |Basic Auth, SSL |Works with the old Ruby Event API (`event['product']['price'] = 10`) # | | - | |Basic Auth, SSL |Works with the new getter/setter APIs (`('[product][price]', 10)`) # | | - | |Basic Auth, SSL |Not compatible with the broker # |========================================================== # # NOTE: We recommended that you use matching Kafka client and broker versions. During upgrades, you should # upgrade brokers before clients because brokers target backwards compatibility. For example, the broker # is compatible with both the consumer and consumer APIs, but not the other way around.
6锛夊崌绾afka_-涓簁afka_- 锛堟垜鐪媖afka-clients-锛屾墍浠ユ病鏈夌敤鏈€鏂扮殑kafka_-锛?/p>
澶ф姝ラ
鍏抽棴鑰乲afka
/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/
澶囦唤鑰侀厤缃枃浠?/p>
鍜?/p>
鍒犻櫎kafka
rm -rf /usr/local/kafka/
rm -rf /data/kafkalogs/*
瀹夎閰嶇疆鏂発afka
wget /apache/kafka//kafka_-
tar zxvf kafka_- -C /usr/local/
ln -s /usr/local/kafka_- /usr/local/kafka
diff浜嗕笅鍜屽彉鍔ㄤ笉澶у彲浠ョ洿鎺ヤ娇鐢?/p>
鍚姩鏂発afka
/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/ &
7)娉ㄦ剰鍑犱釜鍏抽敭閰嶇疆闇€瑕佷慨鏀?/p>
config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
config :group_id, :validate => :string, :default => "logstash"
config :topics, :validate => :array, :default => ["logstash"]
config :consumer_threads, :validate => :number, :default => 1
闄や簡涓婇潰鐨勫嚑涓叧閿厤缃锛宬afka鐨則opic鍒嗙墖淇℃伅闇€瑕侀噸鏂癱reate涓€浠斤紝鍚﹀垯KafkaMonitor鐩戞帶涓嶅嚭Active Topic Consumer鍥惧舰锛屼絾瀹為檯鏄湪宸ヤ綔涓€?/p>