kafka配置

kafka_1.1.1 配置注意事项

1. kafka broker默认配置文件似乎使kafka只监听loopback接口的端口，为了使用kafka监听所有网络接口的对应端口，需要修改默认配置文件，配置文件中关于socket相关的选项有两个，listeners和advertised.listeners，每个选项作用及相互关系需理解。

  ############################# Socket Server Settings #############################
 
 25 # The address the socket server listens on. It will get the value returned from
 26 # java.net.InetAddress.getCanonicalHostName() if not configured.
 27 #   FORMAT:
 28 #     listeners = listener_name://host_name:port
 29 #   EXAMPLE:
 30 #     listeners = PLAINTEXT://your.host.name:9092
 31 #listeners=PLAINTEXT://:9092
 32
 33 # Hostname and port the broker will advertise to producers and consumers. If not set,
 34 # it uses the value for "listeners" if configured.  Otherwise, it will use the value
 35 # returned from java.net.InetAddress.getCanonicalHostName().
 36 #advertised.listeners=PLAINTEXT://your.host.name:9092
 37
 38 # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
 39 #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
 40
 41 # The number of threads that the server uses for receiving requests from the network and sending responses to the network
 42 num.network.threads=3
 43
 44 # The number of threads that the server uses for processing requests, which may include disk I/O
 45 num.io.threads=8
 46
 47 # The send buffer (SO_SNDBUF) used by the socket server
 48 socket.send.buffer.bytes=102400
 49
 50 # The receive buffer (SO_RCVBUF) used by the socket server
 51 socket.receive.buffer.bytes=102400
 52
 53 # The maximum size of a request that the socket server will accept (protection against OOM)
 54 socket.request.max.bytes=104857600

在同一台机器上启动多个broker实例（伪分布式），应该为每个broker实例分配一个配置文件，可使用默认文件副本，根据kafka quick start教程，当中某些项必须修改。包括broker.id、listeners、log.dirs等。

kafka自带java客户端常用命令

1.启动服务器
#启动zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
#启动Kafka
bin/kafka-server-start.sh config/server.properties

2.创建主题
#创建
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
#查看所有主题
bin/kafka-topics.sh --list --zookeeper localhost:2181

3.发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

4.启动消费者
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

5.设置多broker的Kafka集群
#为每个broker创建配置文件
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
#编辑这些配置文件
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1

config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2
#启动这两个节点
bin/kafka-server-start.sh config/server-1.properties &
bin/kafka-server-start.sh config/server-2.properties &
#创建一个新主题，副本为3
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

#停止进程节点
ps aux | grep server-1.properties
kill -9 xxxx

gwet curl ??

ps aux | grep server-1.properties