关闭

Kafka安装部署

标签: kafka集群搭建大数据数据收集kafka
2994人阅读 评论(0) 收藏 举报
分类:

系统环境

组件 版本
CentOS 6.5 64x
zookeeper 3.4.5
kafka 2.10-0.8.1.1

单节点安装

下载kafka并且解压

tar zxvf kafka_2.10-0.8.1.1.tar.gz 
cd kafka_2.10-0.8.1.1/

启动kafka默认配置

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

创建 topic 名为 “test”

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

#列出 topic 
./bin/kafka-topics.sh --list --zookeeper localhost:2181

创建客户端

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

创建消费端

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

集群配置

修改server-1.properties中的参数

cp config/server.properties config/server-1.properties

主要修改内容

broker.id=0
log.dirs=/home/hadoop/development/src/kafka_2.10-0.8.1.1/logs
zookeeper.connect=canbot130:2181,canbot131:2181,canbot132:2181

修改完成后 copy 到其他节点

scp -r ./kafka_2.10-0.8.1.1/ hadoop@canbot131:/home/hadoop/development/src/
scp -r ./kafka_2.10-0.8.1.1/ hadoop@canbot132:/home/hadoop/development/src/

copy完以后需要修改再修改 server-1.properties 中的 broker.id

broker.id=0 192.169.2.130
broker.id=1 192.169.2.131
broker.id=2 192.169.2.132

启动Kafka

分别在canbot130/1/2三个节点都启动

./kafka_2.10-0.8.1.1/bin/kafka-server-start.sh ./kafka_2.10-0.8.1.1/config/server-1.properties &

创建集群 Topic

[hadoop@canbot130 kafka_2.10-0.8.1.1]$./bin/kafka-topics.sh --create --zookeeper canbot130:2181 --replication-factor 3 --partitions 1 --topic test

提示以下内容表示创建Topic 成功

Created topic "test".

查看 Topic 列表

[hadoop@canbot130 kafka_2.10-0.8.1.1]$ ./bin/kafka-topics.sh --list --zookeeper canbot130:2181
test
[hadoop@canbot130 kafka_2.10-0.8.1.1]$ 

创建生产者

./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test

使用该命令创建生产者,然后将在canbot132节点上创建 消费者,查看消息是否被消费

创建消费者

在 canbot132 节点上执行

./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test

生产消息==>消费消息

在canbot130节点上的生产者

[hadoop@canbot130 kafka_2.10-0.8.1.1]$ ./bin/kafka-console-producer.sh --broker-list canbot130:9092 --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
"holl"
[2016-05-31 04:27:30,770] INFO Closing socket connection to /192.168.2.130. (kafka.network.Processor)
"hao xiang shi tong bu l haha"
"test kafka"

在canbot132节点上的消费者所产生的信息

[hadoop@canbot132 kafka_2.10-0.8.1.1]$ ./bin/kafka-console-consumer.sh --zookeeper canbot130:2181 --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2016-05-31 04:27:22,328] INFO Closing socket connection to /192.168.2.132. (kafka.network.Processor)
holl
hao xiang shi tong bu l haha
test kafka

错误记录

错误一

java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/1. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
        at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
        at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
        at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
        at kafka.Kafka$.main(Kafka.scala:46)
        at kafka.Kafka.main(Kafka.scala)

解决方法:该错误是由于server.properties 中的broker.id 重复

0
0
查看评论

spark-jdbc No suitable driver found错误处理

错误如图: 代码: object Mysql extends App{ val sparkConf = new SparkConf()//.setAppName("RDDRelation").setMaster("local[1]") ...
  • xiaofei2017
  • xiaofei2017
  • 2017-09-08 15:49
  • 203

Kafka系列2-producer和consumer报错

1. 使用127.0.0.1启动生产和消费进程: 1)启动生产者进程: bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test 输入消息: this is msg 生产者进...
  • kuluzs
  • kuluzs
  • 2016-06-03 11:39
  • 25406

原来kafka的broker.id = -1,即broker.id 取值范围是大于等于-1

kafka的broker.id = -1,即broker.id 取值范围是大于等于-1; broker.id = -2的机器上broker启动失败的日志提醒如下 java.lang.IllegalArgumentException: requirement failed: broker....
  • xiaofang2015
  • xiaofang2015
  • 2018-02-05 16:26
  • 19

kafka报错

1.Error while executing topic command : replication factor: 1 larger than available brokers: 0 17/04/06 09:47:15 ERROR admin.TopicCommand$: kafka.admi...
  • u013013024
  • u013013024
  • 2017-04-06 13:55
  • 8385

kafka在zookeeper中对应目录

Zookeeper Directories The following gives the zookeeper structures and algorithms used for co-ordination between consumers and brokers. Notation When ...
  • beijing20120926
  • beijing20120926
  • 2013-12-02 15:56
  • 4285

[ActiveMQ]java.net.UnknownHostException: localhost

问题描述: 今天在配置jms的时候,修改了jms的配置文件: 但是在启动jms的时候,报了一个错误: Loading message broker from: xbean:activemq.xml INFO BrokerService - ActiveMQ...
  • a19881029
  • a19881029
  • 2012-12-13 09:09
  • 19299

kafka创建topic报错:KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids

同事新搭建了一个kafka集群,在创建topic的时候遇到错误: 执行:./kafka-topics.sh --zookeeper linux1:2181 --create --topic test --replication-factor 1 --partitions 1 报错: rror ...
  • BrotherDong90
  • BrotherDong90
  • 2016-04-08 13:06
  • 9570

使用zookeeper获取brokers的信息

使用zookeeper获取brokers的信息Table of Contents1. 在ArchLinux上安装libzookeeper2. include 头文件3. 现在编写代码4. 运行程序5. 后续要解决1 在ArchLinux上安装libzookeeperyaourt -S li...
  • sheismylife
  • sheismylife
  • 2016-05-02 11:07
  • 6078

有多个broker的例子

-- Start -- 待续 -- 更多参见:Open Source 精萃 -- 声 明:转载请注明出处 -- Last Updated on 2014-08-30 -- Written by ShangBo on 2014-08-30 -- End
  • shangboerds
  • shangboerds
  • 2014-08-30 15:46
  • 6782

<一>Flex(Flex+j2EE)

步骤:1、新建一个Web Project工程,工程名为:flexDemo2、解压blazeds.war的文件,此时就有两个文件夹 META-INF和WEB-INF, 用解压后的WEB-INF直接覆盖项目中的WEB-INF3、为项目增加Flex支持,右键项目名称,点击Flex Project...
  • ProgramerBird
  • ProgramerBird
  • 2015-04-10 17:13
  • 12418
    个人资料
    • 访问:168552次
    • 积分:2334
    • 等级:
    • 排名:第18849名
    • 原创:60篇
    • 转载:27篇
    • 译文:0篇
    • 评论:46条