jdk部署
下载安装包:wget wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.tar.gz -C /var/tmp 解压:tar zxf /var/tmp/jdk-8u121-linux-i586.tar.gz -C /usr/lib/ 配置: echo " export JAVA_HOME=/usr/lib/jdk-8u121-linux-i586 export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib " > /etc/profile.d/jdk.sh |
zookeeper部署
下载安装包:wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.3-beta/zookeeper-3.5.3-beta.tar.gz 解压: tar zxvf /var/tmp/zookeeper-3.5.3-beta.tar.gz -C /data/src/ link -s /data/src/zookeeper-3.5.3-beta /data/service/zookeeper mkdir /data/zk_data && chown zookeeper:hadoop /data/zk_data && chmod 770 /data/zk_data touch /data/zk_data/myid && chown zookeeper:hadoop /data/zk_data && chmod 770 /data/zk_data echo 1|2|3 > /data/zk_data/myid 在各个节点运行 添加配置文件到/data/service/zookeeper/conf/zoo.cfg : clientPort=2181 tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zk_data maxClientCnxns=512 autopurge.purgeInterval=24 autopurge.snapRetainCount=3 server.1=ip:2900:3900 server.2=ip:2900:3900 server.3=ip:2900:3900 |
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.1/kafka_2.11-2.3.1.tgz -C /var/tmp touch /data/config/kafka/server.properties && chown root:hadoop /data/config/kafk/server.properties && chmod 770 /data/config/kafka/server.properties 添加配置文件到server.properties : port=9092 reserved.broker.max.id=0 broker.id.generation.enable=true num.io.threads=60 num.partitions=20 background.threads=20 log.cleaner.threads=5 num.network.threads=48 log.retention.hours=168 num.replica.fetchers=30 log.cleaner.enable=true delete.topic.enable=true queued.max.requests=4096 log.cleanup.policy=delete log.segment.bytes=1073741824 default.replication.factor=3 offsets.retention.minutes=1500 replica.fetch.wait.max.ms=3000 auto.create.topics.enable=false replica.fetch.max.bytes=134217728 num.recovery.threads.per.data.dir=10 log.retention.check.interval.ms=300000 log.flush.interval.ms=10000 log.flush.interval.messages=200000 replica.socket.receive.buffer.bytes=33554432 socket.send.buffer.bytes=33554432 socket.receive.buffer.bytes=33554432 zookeeper.session.timeout.ms=10000 message.max.bytes=16777216 offsets.retention.check.interval.ms=600000 zookeeper.connect= og.dirs=/mnt/storage00/datum/kafka_data |
2.修改JVM参数、启动服务
参照线上集群修改参数: 1.vim /etc/profile.d/kafka-env.sh export KAFKA_MM_OPTS="-server -XX:+UseG1GC -Xms24g -Xmx24g -XX:MetaspaceSize=1024m -XX:ParallelGCThreads=10 -XX:ConcGCThreads=8" export KAFKA_GC_OPT="-verbose.gc -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution-XX:+PrintGCApplicationStoppedTime-XX:+PrintTenuringDistribution -Xloggc:/data/log/kafka/kafka-gc.log -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/log/kafka/java_pid%p.hprof" export KAFKA_HEAP_OPTS="$KAFKA_MM_OPTS $KAFKA_GC_OPT -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=7070 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" 2.vim /etc/profile.d/zk-env.sh export ZK_MM_OPTS="-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms2024m -Xmx2048m -XX:ParallelGCThreads=33 -XX:ConcGCThreads=8 -XX:CMSInitiatingOccupancyFraction=85" export ZK_GC_OPT="-verbose.gc -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution -Xloggc:/data/log/zookeeper/zk-gc.log -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/log/zookeeper/java_pid%p.hprof" 启动服务: zkServer.sh start kafka-server-start.sh -daemon /data/config/kafka/server.properties |
3.测试集群
kafka-topics.sh --zookeeper zookeeper.connect --create --partitions 6 --replication-factor 3 --topic bilibili kafka-topics.sh --zookeeper zookeeper.connect --list kafka-console-producer.sh --broker-list host:ip --topic bilibili kafka-console-consumer.sh --bootstrap-server host:ip --topic bilibili |
注意:
1.配置文件参数以实际任务占比资源为准
2.在修改jvm中,kafka-env.sh中的KAFKA_HEAP_OPTS参数需要在kafka启动时放开,运行中屏蔽;xms和xmx安集群任务启动占用资源设置