关闭

Kafka安装配置(集群模式)

标签: 分布式系统架构 集群
344人阅读 评论(0) 收藏 举报
分类:

kafka下载:https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1/kafka_2.10-0.8.1.tgz

分别在三台服务器上安装kafka:
tar zxvf kafka_2.10-0.8.1.tgz

修改每台服务器的config/server.properties 

broker.id:  唯一,填数字,本文中分别为132/133/134

host.name:唯一,填服务器IP,之前配置时,把中间的'.'给忘写了,导致kafka集群形同虚设(基本只有leader机器在起作用),以及一系列莫名其妙的问题,伤啊

zookeeper.connect=192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181

先启动zookeeper服务: bin/zkServer.sh start (本文中zookeeper为独立安装,具体过程在此不细述)
再在每台机器上执行: bin/kafka-server-start.sh config/server.properties  

你将会看到如下信息:
[plain] view plain copy
 print?在CODE上查看代码片派生到我的代码片
  1. <span style="font-family:Microsoft YaHei;font-size:14px;">[2014-04-02 02:42:14,220] INFO Verifying properties (kafka.utils.VerifiableProperties)  
  2. [2014-04-02 02:42:14,281] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)  
  3. [2014-04-02 02:42:14,281] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)  
  4. [2014-04-02 02:42:14,281] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)  
  5. [2014-04-02 02:42:14,282] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)  
  6. [2014-04-02 02:42:14,282] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)  
  7. [2014-04-02 02:42:14,282] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)  
  8. [2014-04-02 02:42:14,282] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)  
  9. [2014-04-02 02:42:14,283] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)  
  10. [2014-04-02 02:42:14,283] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)  
  11. [2014-04-02 02:42:14,283] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)  
  12. [2014-04-02 02:42:14,283] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)  
  13. [2014-04-02 02:42:14,284] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)  
  14. [2014-04-02 02:42:14,284] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)  
  15. [2014-04-02 02:42:14,284] INFO Property zookeeper.connect is overridden to 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.utils.VerifiableProperties)  
  16. [2014-04-02 02:42:14,284] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)  
  17. [2014-04-02 02:42:14,309] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)  
  18. [2014-04-02 02:42:14,312] INFO [Kafka Server 0], Connecting to zookeeper on 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.server.KafkaServer)  
  19. [2014-04-02 02:42:14,334] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)  
  20. [2014-04-02 02:42:14,348] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)  
  21. [2014-04-02 02:42:14,348] INFO Client environment:host.name=ubuntu (org.apache.zookeeper.ZooKeeper)  
  22. [2014-04-02 02:42:14,348] INFO Client environment:java.version=1.6.0_37 (org.apache.zookeeper.ZooKeeper)  
  23. [2014-04-02 02:42:14,348] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)  
  24. [2014-04-02 02:42:14,348] INFO Client environment:java.home=/etc/local/jdk1.6.0_37/jre (org.apache.zookeeper.ZooKeeper)  
  25. [2014-04-02 02:42:14,349] INFO Client environment:java.class.path=:/usr/local/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/usr/local/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/usr/local/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/usr/local/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/usr/local/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.ZooKeeper)  
  26. [2014-04-02 02:42:14,349] INFO Client environment:java.library.path=/etc/local/jdk1.6.0_37/jre/lib/amd64/server:/etc/local/jdk1.6.0_37/jre/lib/amd64:/etc/local/jdk1.6.0_37/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)  
  27. [2014-04-02 02:42:14,349] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)  
  28. [2014-04-02 02:42:14,349] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)  
  29. [2014-04-02 02:42:14,349] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)  
  30. [2014-04-02 02:42:14,349] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)  
  31. [2014-04-02 02:42:14,349] INFO Client environment:os.version=3.8.0-29-generic (org.apache.zookeeper.ZooKeeper)  
  32. [2014-04-02 02:42:14,349] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)  
  33. [2014-04-02 02:42:14,349] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)  
  34. [2014-04-02 02:42:14,349] INFO Client environment:user.dir=/usr/local/kafka_2.10-0.8.1 (org.apache.zookeeper.ZooKeeper)  
  35. [2014-04-02 02:42:14,352] INFO Initiating client connection, connectString=192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@7ed75415 (org.apache.zookeeper.ZooKeeper)  
  36. [2014-04-02 02:42:14,412] INFO Opening socket connection to server /192.168.40.134:2181 (org.apache.zookeeper.ClientCnxn)  
  37. [2014-04-02 02:42:14,779] INFO Socket connection established to ubuntu.local/192.168.40.134:2181, initiating session (org.apache.zookeeper.ClientCnxn)  
  38. [2014-04-02 02:42:14,805] INFO Session establishment complete on server ubuntu.local/192.168.40.134:2181, sessionid = 0x14521d0059e0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)  
  39. [2014-04-02 02:42:14,811] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)  
  40. [2014-04-02 02:42:15,045] INFO Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)  
  41. [2014-04-02 02:42:15,072] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)  
  42. [2014-04-02 02:42:15,077] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)  
  43. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".  
  44. SLF4J: Defaulting to no-operation (NOP) logger implementation  
  45. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.  
  46. [2014-04-02 02:42:15,170] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)  
  47. [2014-04-02 02:42:15,176] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)  
  48. [2014-04-02 02:42:15,310] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)  
  49. [2014-04-02 02:42:15,387] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)  
  50. [2014-04-02 02:42:15,527] INFO Registered broker 0 at path /brokers/ids/0 with address ubuntu:9092. (kafka.utils.ZkUtils$)  
  51. [2014-04-02 02:42:15,571] INFO [Kafka Server 0], started (kafka.server.KafkaServer)  
  52. [2014-04-02 02:42:15,654] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)  
  53. </span>  

创建topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic mykafka
查看Topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181
显示Topic:mykafka

查看详细信息:
bin/kafktopics.sh --describe --zookeeper 192.168.40.132:2181
Topic:mykafka PartitionCount:1 ReplicationFactor:3 Configs:
  Topic: mykafka Partition: 0 Leader: 133 Replicas: 133,134,132 Isr: 134
发送消息:
bin/kafka-console-producer.sh --broker-list 192.168.40.134:9092 --topic mykafka
23423
^C
bin/kafka-console-producer.sh --brokelist 192.168.40.134:9092 --topic mykafka
4533
如果出现以下信息,则需要下载slftj-nop-1.5.jar,并将其cp至kafka的libs目录下:
  1. <span style="font-family:Microsoft YaHei;font-size:14px;">SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".  
  2. SLF4J: Defaulting to no-operation (NOP) logger implementation  
  3. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. </span>  
接收消息:
bin/kafka-console-consumer.sh --zookeeper 192.168.40.133:2181 --topic mykafka --from-beginning
sfsa
asdg
sfa
asfda
sadf
1
3
4
5
23423
4533
0
0
查看评论

kafka集群安装配置

1,准备三台机器,例如IP地址分别为109.105.111.11, 109.105.111.12, 109.105.111.13的三台机器。 2,下载kafka稳定版本,我下载的是kafka_2.10-0.9.0.0.tgz。 3,分别在三台机器上解压放到想安装的目录,我的目录统一为usr/ss...
  • silent_strings
  • silent_strings
  • 2016-03-13 19:17
  • 3895

Kafka单机、集群模式安装详解(二)

本文环境如下: 操作系统:CentOS 6 32位 JDK版本:1.8.0_77 32位 Kafka版本:0.9.0.1(Scala 2.11) 接上篇 Kafka单机、集群模式安装详解(一)6. 单节点多Broker模式Kafka使用可以有多种模式,包括单节点单Broker,单节...
  • Lnho2015
  • Lnho2015
  • 2016-05-09 14:47
  • 5249

kafka集群安装与配置

一、集群安装 1. Kafka下载: 可以从kafka官方网站(http://kafka.apache.org)上找到下载地址,再wgetwget http://mirrors.cnnic.cn/apache/kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz 解压该...
  • bluejoe2000
  • bluejoe2000
  • 2016-03-14 12:09
  • 647

Centos7 安装Kafka集群

准备好三台主机: Node-131 Node-132 Node-133 下载解压:从官网下载Kafka 安装包,解压安装: tar zxvf kafka_xxx.tgz mv kafka_xxx kafka cd kafka启动Zookeeper集群zookeeper集群搭建参考之前的...
  • sunjiaminaini
  • sunjiaminaini
  • 2017-09-05 15:54
  • 277

centos7下kafka集群搭建

概述集群安装或者单机安装都可以,这里介绍集群安装。Kafka本身安装包也自带了zookeeper,也可以使用其自带的zookeeper。建议试用自己安装的zookeeper,本教程试用单独安装的zookeeper。安装环境3台centos7虚拟机:10.15.21.62 10.10.182.168 ...
  • xuefeng1207
  • xuefeng1207
  • 2017-08-10 15:16
  • 418

Kafka集群模式部署

环境:kafka 0.8.1.1基本概念 Kafka维护按类区分的消息,称为主题(topic) 生产者(producer)向kafka的主题发布消息 消费者(consumer)向主题注册,并且接收发布到这些主题的消息 kafka以一个拥有一台或多台服务器的集群运行着,每一台服务器称为broker z...
  • lsshlsw
  • lsshlsw
  • 2015-08-05 10:39
  • 2154

Kafka集群搭建详细步骤

Kafka集群搭建 1、 Kafka的安装需要java环境,cent os 7自带java1.6版本,可以不用重新安装,直接使用自带的jdk 即可;如果觉得jdk版本太旧,也可以自己重新安装; 2、 准备好kafka安装包,官网下载地址: ...
  • zxy987872674
  • zxy987872674
  • 2017-05-18 16:45
  • 11588

Kafka单机、集群模式安装详解(一)

本文环境如下: 操作系统:CentOS 6 32位 JDK版本:1.8.0_77 32位 Kafka版本:0.9.0.1(Scala 2.11) 1. 所需要的环境Kafka需要以下运行环境: Java 安装参考CentOS 6使用rpm方式安装JDK8 Zookeeper ...
  • Lnho2015
  • Lnho2015
  • 2016-05-09 14:41
  • 16616

kafka安装配置(集群)以及收发消息讲解

kafka安装配置(集群)以及收发消息讲解 一、安装配置 1、下载解压 wget    http://mirror.bit.edu.cn/apache/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz tar   -zxv...
  • chenxun2009
  • chenxun2009
  • 2017-03-07 22:50
  • 1331

ELK5.3+Kafka集群配置

【一】资源准备 # 3台4C*8G, 安装Zookeeper、Kafka、Logstash——Broker(input: filebeat; output: Kafka)   10.101.2.23 10.101.2.24 10.101.2.25 # 2台4C*8G,...
  • gb4215287
  • gb4215287
  • 2017-08-22 17:37
  • 317
    个人资料
    • 访问:11793次
    • 积分:234
    • 等级:
    • 排名:千里之外
    • 原创:3篇
    • 转载:49篇
    • 译文:0篇
    • 评论:0条