kafka安装，管理，以及常见问题的解决【如：服务端是ok的，java客户端却始终调不通】...

2019独角兽企业重金招聘Python工程师标准>>>

安装步骤：

下载     http://kafka.apache.org/downloads.html

解压      

tar -zxvf kafka_2.10-0.8.1.1.tgz 
cd kafka_2.10-0.8.1.1

启动服务：  首先启动zookeeper服务      

 bin/zookeeper-server-start.sh config/zookeeper.properties &    

启动Kafka  

bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 & 

创建topic         创建一个"test"的topic，一个分区一个副本        

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看主题       

bin/kafka-topics.sh --list --zookeeper localhost:2181

查看主题详情      

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test 

删除主题      

bin/kafka-run-class.sh kafka.admin.TopicCommand –delete --topic test --zookeeper 10.1.10.77:2181

 

Kafka学习整理八(topic管理) 添加，修改分区等

创建生产者 producer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

创建消费者 consumer从头开始消费

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

创建消费者 consumer从最新消费

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

增加分区数

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name--partitions 40

增加配置

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --config flush.messages=1

删除配置

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --delete-config flush.messages=1

 

(四)、删除topic

目前删除操作在默认情况下只是打上一个删除的标记，在重新启动kafka 后才删除。如果需要立即删除，则需要在server.properties中配置

delete.topic.enable=true

 

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --delete --topic my_topic_name

 

问题一：记得第一次安装时候，java客户端却始终调不通，调了很久，一直不通，都是是按网上步骤一步一步安装的，安装也没有报错，直接在linux上面通过以下命、

创建生产者 producer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

服务端是ok的，能够发送消息，

创建消费者 consumer

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

这个也能收到消息

但是java 客户端就是调不通，重试3次就抛异常了

解决方案：

 编辑 这个文件打开下面红色部分加上去，重启kafka就解决啦

[root@localhost kafka_2.11-0.10.1.0]# vi ./config/server.properties  vi ./config/server.properties 

#listeners=PLAINTEXT://:9092
steners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://120.77.155.63:9092

参考链接：http://www.javacoder.cn/?p=849&utm_source=tuicool&utm_medium=referral

问题二：kafka跑一端时间【一般2个小时】就挂了，kafka进程还在，但是发送不了数据，也消费不了，就好像阻塞了一样，zookeeper没有挂

错误信息如下：

kafka.common.KafkaException: Controller doesn't exist
	at kafka.utils.ZkUtils.getController(ZkUtils.scala:153)
	at kafka.server.KafkaServer.networkClientControlledShutdown$1(KafkaServer.scala:373)
	at kafka.server.KafkaServer.kafka$server$KafkaServer$$controlledShutdown(KafkaServer.scala:522)
	at kafka.server.KafkaServer$$anonfun$shutdown$1.apply$mcV$sp(KafkaServer.scala:544)
	at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:76)
	at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
	at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:47)
	at kafka.utils.Logging$class.swallow(Logging.scala:94)
	at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:47)
	at kafka.server.KafkaServer.shutdown(KafkaServer.scala:544)
	at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:49)
	at kafka.Kafka$$anon$1.run(Kafka.scala:63)
[2016-12-15 19:53:08,006] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2016-12-15 19:53:08,011] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2016-12-15 19:53:08,011] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2016-12-15 19:53:08,012] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2016-12-15 19:53:08,014] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,020] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,020] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,108] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)
[2016-12-15 19:53:08,108] INFO [Replica Manager on Broker 0]: Shutting down (kafka.server.ReplicaManager)
[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2016-12-15 19:53:08,109] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,577] INFO [Replica Manager on Broker 0]: Shut down completely (kafka.server.ReplicaManager)
[2016-12-15 19:53:08,578] INFO Shutting down. (kafka.log.LogManager)
[2016-12-15 19:53:08,813] INFO re-registering broker info in ZK for broker 0 (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,813] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,855] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,869] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(10.1.10.80,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2016-12-15 19:53:08,869] INFO done re-registering broker (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,869] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,881] INFO Shutdown complete. (kafka.log.LogManager)
[2016-12-15 19:53:08,882] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.GroupCoordinator)
[2016-12-15 19:53:08,882] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.GroupCoordinator)
[2016-12-15 19:53:09,512] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)

解决方案如下：

把之前的启动方式改成后台守护进程的方式启动，后面就没有再挂了。

./bin/kafka-server-start.sh -daemon config/server.properties

 

如有其它问题可以留言或email我wei.liu2021@gmail.com

转载于:https://my.oschina.net/u/1266221/blog/798782