Kafka 集群安装脚本

版权声明:本文为博主编写文章,未经博主允许转载,转载请注明出处。 https://blog.csdn.net/u012373815/article/details/79135270

最近又在搞Kafka集群,安装kafka集群感觉太麻烦,所以就写了一个简单的kafka 集群安装脚本,但是有个前提,各个机器的 ssh 免密要配置好,然后jdk 要安装成功。

kafka 安装分两步,一是安装 zookkper ,二是安装配置集群。

注意:
* 各个机器的 ssh 免密要配置好,jdk 要安装成功。
* 如果安装在本机,那么本机也要配置ssh自免密。
* 脚本要和安装包在同一个目录。

zookeeper 安装脚本

ROOT:为安装的根目录
USER :为安装的用户
ZOOKEEPER_NAME:为安装包的名字

#!bin/sh

ROOT=/usr/cpic/testapps
USER=cpic

ZOOKEEPER_NAME=zookeeper-3.4.10
ZOOKEEPER_TAR=$ZOOKEEPER_NAME.tar.gz

NODES="192.168.100.90"

function install_zookeeper()
{
    for i in $NODES
    do  
        ssh -oStrictHostKeyChecking=no $USER@${i} mkdir -p $ROOT/$ZOOKEEPER_NAME
        scp -rp $ZOOKEEPER_NAME $USER@${i}:$ROOT
        ssh $USER@${i} "cd $ROOT/$ZOOKEEPER_NAME/conf;cp zoo_sample.cfg zoo.cfg";
    done
    echo "success install $ZOOKEEPER_NAME";
exit 0  
}

# 判断zookeeper tar.gz  包是否存在,存在则安装
if [ -f "$ZOOKEEPER_TAR" ]; then
     echo "开始解压 $ZOOKEEPER_TAR 。。。。";
     tar -zxvf $ZOOKEEPER_TAR;
     echo "开始安装zookeeper 。。。。";
     install_zookeeper;
else
    echo "$ZOOKEEPER_TAR 在当前目录下不存在";
fi

Kafka安装脚本

ROOT:为安装的根目录
USER :为安装的用户
KAFKA_NAME:为安装包的名字

#!bin/sh

DIR=/usr/cpic/testapps
ZOOKEEPER_HOST_PORT=192.168.100.90:2181
PARTITIONS_NUM=3
BROKER_ID=0
USER=cpic

KAFKA_NAME=kafka_2.11-0.10.1.0
KAFKA_TAR=$KAFKA_NAME.tgz



BROKERS="192.168.100.91 192.168.100.92 192.168.100.93"


function install_kafka()
{
    for i in $BROKERS
    do
        ssh -oStrictHostKeyChecking=no $USER@${i} mkdir -p $DIR/data/kaka-logs
        scp -rp $KAFKA_NAME $USER@${i}:$DIR
        echo modify @${i} server.properties 
        ssh $USER@${i} "cd $DIR/$KAFKA_NAME/config;sed -i 's?^log.dirs=?#log.dirs=?' server.properties ;sed -i  's?^zookeeper.connect=?#zookeeper.connect=?' server.properties ;sed -i  's?^num.partitions=?#num.partitions=?' server.properties;sed -i  's?^broker.id=?#broker.id=?' server.properties ;echo “” >> server.properties;echo "broker.id=$BROKER_ID" >> server.properties;echo "log.dirs=$DIR/kafka-logs" >> server.properties;echo "zookeeper.connect=$ZOOKEEPER_HOST_PORT" >> server.properties;echo "num.partitions=$PARTITIONS_NUM" >> server.properties ;echo "port=9092" >> server.properties ;echo "host.name=${i}" >> server.properties;"
        let BROKER_ID+=1;
    done
    echo success modify all server.properties 
exit 0  
}

# 判断kafka tar.gz  包是否存在,存在则安装
if [ -f "$KAFKA_TAR" ]; then
     echo "开始解压 $KAFKA_NAME.tgz 。。。。"
     tar -zxvf $KAFKA_TAR;
     echo "开始安装Kafka 。。。。";
     install_kafka;
 else
    echo "$KAFKA_TAR 在当前目录下不存在";
fi

关于Kafka 的集群启动停止 请看:Kafka 集群启停脚本

kafka生产者连接集群kafka集群失败

11-12

报错信息如下:rnException in thread "Thread-0" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.rn at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)rn at kafka.producer.Producer.send(Unknown Source)rn at kafka.javaapi.producer.Producer.send(Unknown Source)rn at testKafka.KafkaProducer.run(KafkaProducer.java:29)rn下边是生产者源码(采用同步模式):rnimport java.util.Properties;rnimport java.util.concurrent.TimeUnit;rnrnimport kafka.javaapi.producer.Producer;rnimport kafka.producer.KeyedMessage;rnimport kafka.producer.ProducerConfig;rnimport kafka.serializer.StringEncoder;rnrn/**rn * KAFKA生产者编程模型:同步,异步rn * rn */rnpublic class KafkaProducer extends Thread rnrn private String topic;rnrn public KafkaProducer(String topic) rn super();rn this.topic = topic;rn rnrn @Overridern public void run()rn Producer producer = createProducer(); rn int i=0; rn while(true) rn producer.send(new KeyedMessage(topic, "message: " + i++)); rn try rn TimeUnit.SECONDS.sleep(1); rn catch (InterruptedException e) rn e.printStackTrace(); rn rn rn rnrn //创建生产者对象,采用同步的消息传送方式rn private Producer createProducer() rn Properties properties = new Properties();rn //配置zk,主机地址+zk端口号rn properties.put("zookeeper.connect","10.10.31.120:2181,10.10.31.121:2181,10.10.31.122:2181");rn properties.put("serializer.class",StringEncoder.class.getName());rn //配置kafka broker集群rn properties.put("metadata.broker.list","10.10.31.120:9092,10.10.31.121:9092,10.10.31.122:9092");rnrn properties.put("request.required.acks","1");rn rn return new Producer(new ProducerConfig(properties));rn rnrn public static void main(String[] args) rn KafkaProducer kp = new KafkaProducer("topic-hahaha");rn kp.start();rn rnrnrn

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试