Kafka组件connector使用详解

1、文件准备：

应用程序=>kafka.connector.hbase.jar

日志配置=>connect-log4j.properties

Work启动配置=>connect-distributed.properties

 

2、文件

 日志配置：

log4j.rootLogger=INFO, stdout,debug

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR

log4j.logger.org.I0Itec.zkclient=ERROR

log4j.appender.debug=org.apache.log4j.DailyRollingFileAppender

log4j.appender.debug.File=/opt/data/kafka/2.11-1.1.0/logs2/da.log

log4j.appender.debug.MaxFileSize=1GB

log4j.appender.debug.MaxBackupIndex=100

log4j.appender.debug.Threshold = DEBUG

log4j.appender.debug.DatePattern='.'yyyy-MM-dd-HH

log4j.appender.debug.layout=org.apache.log4j.PatternLayout

log4j.appender.debug.layout.ConversionPattern=[%d{yyyy-MM-dd HH:mm:ss}][(%F:%L)] %p %m%n

 

集群配置：

bootstrap.servers=10.200.0.0:9092,10.200.0.0:9092

group.id=connect-cluster

key.converter=org.apache.kafka.connect.storage.StringConverter

value.converter=org.apache.kafka.connect.storage.StringConverter

internal.key.converter=org.apache.kafka.connect.json.JsonConverter

internal.value.converter=org.apache.kafka.connect.json.JsonConverter

rest.host.name=10.200.0.0

rest.port=8083

 

程序：

继承sinkTask接口并重写put方法进行数据流处理

 

3、部署

将文件置于connect_customer_sink目录下

export KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/opt/applications/kafka/2.11-0.10.1.0/connect_customer_sink/connect-log4j.properties
export CLASSPATH=/opt/applications/kafka/2.11-0.10.1.0/connect_customer_sink/kafka.connector.hbase.jar
./connect-distributed.sh -daemon ../connect_customer_sink/connect-distributed.properties

利用rest方式添加connector以及配置文件

curl -X POST  127.0.0.1:10086/connectors -H "Content-Type:application/json"   -d '{"name":"sink-connector","config":{"connector.class":"kafka.connect.hbase.sink.HBaseSinkConnector","tasks.max":"10","topics":"DWH3,AUDIT,APPLY,CMPROD,LEND,P2P,USER","hbase.zookeeper.quorum":"127.0.0.3:2181,127.0.0.2:2181,127.0.0.1:2181"}}'

curl -X GET  10.20.0.134:10086/connectors

curl -X DELETE 10.20.0.134:10086/connectors/connect-sink-connector

 

注:connector消费是手动提交