Kafka配置内外网分流

Kafka配置内外网分流

背景：
之前服务器一直有两块网卡，一块千兆网卡，一块万兆网卡。由于之前数据量不算大，所以直接使用了千兆网卡，并配置了外网映射。随着数据量增大，kafka集群内部通信时网络带宽一直居高不下，甚至将千兆网跑满，导致数据积压。这时候不得不配置内外网分流，将内网通信流量全部走万兆网卡，千兆网卡对外提供服务

1. 千兆网kafka配置文件

cat server.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

broker.id=3
listeners=SASL_PLAINTEXT://10.10.1.119:9092
advertised.listeners=SASL_PLAINTEXT://10.10.1.119:9092
security.inter.broker.protocol=SASL_PLAINTEXT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka

num.partitions=3
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1



log.retention.hours=168

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connect=10.10.1.117:2181,10.10.1.118:2181,10.10.1.119:2181
zookeeper.connection.timeout.ms=6000

group.initial.rebalance.delay.ms=0
broker.id.generation.enable=false
reserved.broker.max.id=10000
super.users=User:admin
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
allow.everyone.if.no.acl.found=false

2. 万兆网kafka配置文件

内网IP（万兆网卡）：10.10.2.57
外网IP（千兆网卡）：10.10.1.57
cat server.properties

broker.id=11
#security.inter.broker.protocol=SASL_PLAINTEXT
listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
listeners=INTERNAL://10.10.2.57:19092,EXTERNAL://10.10.1.57:9092
advertised.listeners=INTERNAL://10.10.2.57:19092,EXTERNAL://221.30.9.12:9092
inter.broker.listener.name=INTERNAL
log.dirs=/data/raid10/local-volume/kafka
zookeeper.connect=10.10.1.57:2181,10.10.1.58:2181,10.10.1.59:2181
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
num.partitions=12
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
default.replication.factor=3
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.retention.bytes=-1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
group.max.session.timeout.ms=600000
request.timeout.ms=60000
message.max.byte=5242880
replica.fetch.max.bytes=5242880
broker.id.generation.enable=false
reserved.broker.max.id=10000
auto.create.topics.enable=true
super.users=User:admin
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
replica.lag.time.max.ms=60000

主要参数：

listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
listeners=INTERNAL://10.10.24.57:19092,EXTERNAL://10.10.22.57:9092
advertised.listeners=INTERNAL://10.10.24.57:19092,EXTERNAL://221.30.9.12:9092
inter.broker.listener.name=INTERNAL

参数解释：
listener.security.protocol.map：配置kafka认证协议，我们是用的SASL_PLAINTEXT
listeners：配置内网和外网监听地址；INTERNAL内网，EXTERNAL外网
advertised.listeners：公开的监听地址，表示客户端可以通过ip:port连接kafka，由于kafka需要公网请求访问，所以EXTERNAL必须要配置成公网映射的IP和端口
inter.broker.listener.name：kafka默认使用的监听地址，表示默认使用内网监听地址通信