kafka安装java,kafka安装，管理，以及常见问题的解决【如：服务端是ok的，java客户端却始终调不通】...

安装步骤：

下载     http://kafka.apache.org/downloads.html

解压

tar -zxvf kafka_2.10-0.8.1.1.tgz

cd kafka_2.10-0.8.1.1

启动服务：  首先启动zookeeper服务

bin/zookeeper-server-start.sh config/zookeeper.properties &

启动Kafka

bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &

创建topic         创建一个"test"的topic，一个分区一个副本

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看主题

bin/kafka-topics.sh --list --zookeeper localhost:2181

查看主题详情

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

删除主题

bin/kafka-run-class.sh kafka.admin.TopicCommand –delete --topic test --zookeeper 10.1.10.77:2181

Kafka学习整理八(topic管理) 添加，修改分区等

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

增加分区数

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name--partitions 40

增加配置

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --config flush.messages=1

删除配置

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --delete-config flush.messages=1

(四)、删除topic

目前删除操作在默认情况下只是打上一个删除的标记，在重新启动kafka 后才删除。如果需要立即删除，则需要在server.properties中配置

delete.topic.enable=true

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --delete --topic my_topic_name

问题一：记得第一次安装时候，java客户端却始终调不通，调了很久，一直不通，都是是按网上步骤一步一步安装的，安装也没有报错，直接在linux上面通过以下命、

创建生产者 producer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

服务端是ok的，能够发送消息，

创建消费者 consumer

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

这个也能收到消息

但是java 客户端就是调不通，重试3次就抛异常了

解决方案：

编辑 这个文件打开下面红色部分加上去，重启kafka就解决啦

[root@localhost kafka_2.11-0.10.1.0]# vi ./config/server.properties vi ./config/server.properties

#listeners=PLAINTEXT://:9092

steners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,

# it uses the value for "listeners" if configured. Otherwise, it will use the value

# returned from java.net.InetAddress.getCanonicalHostName().

#advertised.listeners=PLAINTEXT://your.host.name:9092

advertised.listeners=PLAINTEXT://120.77.155.63:9092

问题二：kafka跑一端时间【一般2个小时】就挂了，kafka进程还在，但是发送不了数据，也消费不了，就好像阻塞了一样，zookeeper没有挂

错误信息如下：

kafka.common.KafkaException: Controller doesn't exist

at kafka.utils.ZkUtils.getController(ZkUtils.scala:153)

at kafka.server.KafkaServer.networkClientControlledShutdown$1(KafkaServer.scala:373)

at kafka.server.KafkaServer.kafka$server$KafkaServer$$controlledShutdown(KafkaServer.scala:522)

at kafka.server.KafkaServer$$anonfun$shutdown$1.apply$mcV$sp(KafkaServer.scala:544)

at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:76)

at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)

at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:47)

at kafka.utils.Logging$class.swallow(Logging.scala:94)

at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:47)

at kafka.server.KafkaServer.shutdown(KafkaServer.scala:544)

at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:49)

at kafka.Kafka$$anon$1.run(Kafka.scala:63)

[2016-12-15 19:53:08,006] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)

[2016-12-15 19:53:08,011] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)

[2016-12-15 19:53:08,011] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)

[2016-12-15 19:53:08,012] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)

[2016-12-15 19:53:08,014] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,020] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2016-12-15 19:53:08,020] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)

[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-15 19:53:08,108] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)

[2016-12-15 19:53:08,108] INFO [Replica Manager on Broker 0]: Shutting down (kafka.server.ReplicaManager)

[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)

[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)

[2016-12-15 19:53:08,109] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:08,577] INFO [Replica Manager on Broker 0]: Shut down completely (kafka.server.ReplicaManager)

[2016-12-15 19:53:08,578] INFO Shutting down. (kafka.log.LogManager)

[2016-12-15 19:53:08,813] INFO re-registering broker info in ZK for broker 0 (kafka.server.KafkaHealthcheck$SessionExpireListener)

[2016-12-15 19:53:08,813] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)

[2016-12-15 19:53:08,855] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2016-12-15 19:53:08,869] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(10.1.10.80,9092,PLAINTEXT) (kafka.utils.ZkUtils)

[2016-12-15 19:53:08,869] INFO done re-registering broker (kafka.server.KafkaHealthcheck$SessionExpireListener)

[2016-12-15 19:53:08,869] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck$SessionExpireListener)

[2016-12-15 19:53:08,881] INFO Shutdown complete. (kafka.log.LogManager)

[2016-12-15 19:53:08,882] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.GroupCoordinator)

[2016-12-15 19:53:08,882] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2016-12-15 19:53:09,157] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.GroupCoordinator)

[2016-12-15 19:53:09,512] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)

解决方案如下：

把之前的启动方式改成后台守护进程的方式启动，后面就没有再挂了。

./bin/kafka-server-start.sh -daemon config/server.properties

如有其它问题可以留言或email我wei.liu2021@gmail.com