安装步骤:
下载 http://kafka.apache.org/downloads.html
解压
tar -zxvf kafka_2.10-0.8.1.1.tgz
cd kafka_2.10-0.8.1.1
启动服务: 首先启动zookeeper服务
bin/zookeeper-server-start.sh config/zookeeper.properties &
启动Kafka
bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
创建topic 创建一个"test"的topic,一个分区一个副本
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
查看主题
bin/kafka-topics.sh --list --zookeeper localhost:2181
查看主题详情
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
删除主题
bin/kafka-run-class.sh kafka.admin.TopicCommand –delete --topic test --zookeeper 10.1.10.77:2181
Kafka学习整理八(topic管理) 添加,修改分区等
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
创建消费者 consumer从头开始消费
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
增加分区数
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name--partitions 40
增加配置
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --config flush.messages=1
删除配置
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic my_topic_name --delete-config flush.messages=1
(四)、删除topic
目前删除操作在默认情况下只是打上一个删除的标记,在重新启动kafka 后才删除。如果需要立即删除,则需要在server.properties中配置
delete.topic.enable=true
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --delete --topic my_topic_name
问题一:记得第一次安装时候,java客户端却始终调不通,调了很久,一直不通,都是是按网上步骤一步一步安装的,安装也没有报错,直接在linux上面通过以下命、
创建生产者 producer
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
服务端是ok的,能够发送消息,
创建消费者 consumer
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
这个也能收到消息
但是java 客户端就是调不通,重试3次就抛异常了
解决方案:
编辑 这个文件打开下面红色部分加上去,重启kafka就解决啦
[root@localhost kafka_2.11-0.10.1.0]# vi ./config/server.properties vi ./config/server.properties
#listeners=PLAINTEXT://:9092
steners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://120.77.155.63:9092
参考链接:http://www.javacoder.cn/?p=849&utm_source=tuicool&utm_medium=referral
问题二:kafka跑一端时间【一般2个小时】就挂了,kafka进程还在,但是发送不了数据,也消费不了,就好像阻塞了一样,zookeeper没有挂
错误信息如下:
kafka.common.KafkaException: Controller doesn't exist
at kafka.utils.ZkUtils.getController(ZkUtils.scala:153)
at kafka.server.KafkaServer.networkClientControlledShutdown$1(KafkaServer.scala:373)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$controlledShutdown(KafkaServer.scala:522)
at kafka.server.KafkaServer$$anonfun$shutdown$1.apply$mcV$sp(KafkaServer.scala:544)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:76)
at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:47)
at kafka.utils.Logging$class.swallow(Logging.scala:94)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:47)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:544)
at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:49)
at kafka.Kafka$$anon$1.run(Kafka.scala:63)
[2016-12-15 19:53:08,006] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2016-12-15 19:53:08,011] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2016-12-15 19:53:08,011] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2016-12-15 19:53:08,012] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2016-12-15 19:53:08,014] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,020] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,020] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Produce], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,101] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,107] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-15 19:53:08,108] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)
[2016-12-15 19:53:08,108] INFO [Replica Manager on Broker 0]: Shutting down (kafka.server.ReplicaManager)
[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2016-12-15 19:53:08,109] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2016-12-15 19:53:08,109] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,120] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:08,577] INFO [Replica Manager on Broker 0]: Shut down completely (kafka.server.ReplicaManager)
[2016-12-15 19:53:08,578] INFO Shutting down. (kafka.log.LogManager)
[2016-12-15 19:53:08,813] INFO re-registering broker info in ZK for broker 0 (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,813] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,855] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-15 19:53:08,869] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(10.1.10.80,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2016-12-15 19:53:08,869] INFO done re-registering broker (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,869] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck$SessionExpireListener)
[2016-12-15 19:53:08,881] INFO Shutdown complete. (kafka.log.LogManager)
[2016-12-15 19:53:08,882] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.GroupCoordinator)
[2016-12-15 19:53:08,882] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,002] INFO [ExpirationReaper-0], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [ExpirationReaper-0], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-15 19:53:09,157] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.GroupCoordinator)
[2016-12-15 19:53:09,512] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
解决方案如下:
把之前的启动方式改成后台守护进程的方式启动,后面就没有再挂了。
./bin/kafka-server-start.sh -daemon config/server.properties
如有其它问题可以留言或email我wei.liu2021@gmail.com