kafka系列之-linux系统，kafka服务端安装

最新推荐文章于 2023-08-21 18:44:41 发布
最新推荐文章于 2023-08-21 18:44:41 发布
阅读量576 收藏
点赞数 1
分类专栏： Linux|windows系统 # Kafka
本文为博主原创文章，未经博主允许不得转载。
本文链接：https://blog.csdn.net/weinichendian/article/details/103680554
版权

1.服务端安装包下载

下载路径，我下载的是Scala 2.11版本。
在这里插入图片描述

2.上传安装包到服务器。可以用ftp。

在这里插入图片描述
在这里插入图片描述

3.解压缩。

[root@instance-n4r06itt middleware]# tar zxvf kafka_2.11-1.1.0.tgz

4.创建日志文件夹。

[root@instance-n4r06itt kafka_2.11-1.1.0]# mkdir logs
[root@instance-n4r06itt kafka_2.11-1.1.0]# ll
total 56
drwxr-xr-x 3 root root  4096 Mar 24  2018 bin
drwxr-xr-x 2 root root  4096 Mar 24  2018 config
drwxr-xr-x 2 root root  4096 Dec 24 11:43 libs
-rw-r--r-- 1 root root 28824 Mar 24  2018 LICENSE
drwxr-xr-x 2 root root  4096 Dec 24 11:43 logs
-rw-r--r-- 1 root root   336 Mar 24  2018 NOTICE
drwxr-xr-x 2 root root  4096 Mar 24  2018 site-docs

5.修改配置文件。

[root@instance-n4r06itt config]# pwd
/opt/middleware/kafka_2.11-1.1.0/config
[root@instance-n4r06itt config]# vim server.properties

由于我只有一台服务器，没有用虚拟机，先模拟单机模式。主要配置topic是否可以删除、kafka监听地址、zookeeper本地就可以、日志路径。

#是否允许删除topic，默认false不能手动删除
delete.topic.enable=true

#当前kafka服务侦听的地址和端口，端口默认是9092
listeners = PLAINTEXT://106.12.175.83:9092

# A comma separated list of directories under which to store log files
log.dirs=/opt/middleware/kafka_2.11-1.1.0/logs

# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

6.添加环境变量。

kafka安装路径保存一下，/opt/middleware/kafka_2.11-1.1.0

[root@instance-n4r06itt config]# vim /etc/profile

#KAFKA_HOME
export KAFKA_HOME=/opt/middleware/kafka_2.11-1.1.0
export PATH=$PATH:$KAFKA_HOME/bin

保存使其立即生效

[root@instance-n4r06itt config]# source /etc/profile

7.启动zookeeper。

zookeeper系列之-linux系统服务安装

[root@instance-n4r06itt bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/middleware/zookeeper-3.4.8/bin/../conf/zoo.cfg
Mode: standalone

8.启动kafka。

[root@instance-n4r06itt bin]# pwd
/opt/middleware/kafka_2.11-1.1.0/bin
[root@instance-n4r06itt bin]# ./kafka-server-start.sh ../config/server.properties &

出现异常

kafka.common.KafkaException: Socket server failed to bind to 106.12.175.83:9092: Cannot assign requested address.

在这里插入图片描述这是我自己配置的一个服务器地址，怀疑是不是端口没有开放的原因。但是实际原因不是端口的问题，我之前配置的106.12.175.83是我百度云服务器的外网地址。我们需要拿到linux本地inet

[root@instance-n4r06itt bin]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.16.4  netmask 255.255.240.0  broadcast 192.168.31.255
        inet6 fe80::f816:3eff:fea7:c7de  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:a7:c7:de  txqueuelen 1000  (Ethernet)
        RX packets 3215936  bytes 626854111 (597.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2958679  bytes 627972940 (598.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 891689  bytes 554427225 (528.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 891689  bytes 554427225 (528.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

然后重新修改配置文件

[root@instance-n4r06itt config]# vim server.properties

在这里插入图片描述
再次重新启动kafka

[root@instance-n4r06itt bin]# ./kafka-server-start.sh ../config/server.properties &

[2019-12-24 14:57:49,519] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-12-24 14:57:49,522] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-12-24 14:57:49,545] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-12-24 14:57:49,596] INFO Loading logs. (kafka.log.LogManager)
[2019-12-24 14:57:49,616] INFO Logs loading complete in 20 ms. (kafka.log.LogManager)
[2019-12-24 14:57:49,639] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2019-12-24 14:57:49,653] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2019-12-24 14:57:50,700] INFO Awaiting socket connections on 192.168.16.4:9092. (kafka.network.Acceptor)
[2019-12-24 14:57:50,826] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2019-12-24 14:57:50,867] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:50,872] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:50,884] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:50,920] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-12-24 14:57:51,028] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-12-24 14:57:51,042] INFO Result of znode creation at /brokers/ids/0 is: OK (kafka.zk.KafkaZkClient)
[2019-12-24 14:57:51,043] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(192.168.16.4,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2019-12-24 14:57:51,045] WARN No meta.properties file under dir /opt/middleware/kafka_2.11-1.1.0/logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-12-24 14:57:51,263] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:51,276] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:51,308] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-12-24 14:57:51,320] INFO Creating /controller (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-12-24 14:57:51,333] INFO Result of znode creation at /controller is: OK (kafka.zk.KafkaZkClient)
[2019-12-24 14:57:51,360] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2019-12-24 14:57:51,392] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2019-12-24 14:57:51,413] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 2 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-12-24 14:57:51,476] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-12-24 14:57:51,592] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-12-24 14:57:51,636] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-12-24 14:57:51,664] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-12-24 14:57:51,828] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-12-24 14:57:51,873] INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-24 14:57:51,874] INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-24 14:57:51,876] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

在这里插入图片描述kafka启动成功。

9.基本使用。

9.1 创建Topic
zookeeper 为zk集群地址，使用任意一个节点都行。
replication-factor 为复制的份数，Kafka实际上会将一个消息复制多份存储，保证不丢失。
partitions 分区，真实的物理节点。

[root@instance-n4r06itt kafka_2.11-1.1.0]# bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic my-test
Created topic "my-test".
[2019-12-24 15:15:44,092] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions my-test-0 (kafka.server.ReplicaFetcherManager)
[2019-12-24 15:15:44,182] INFO [Log partition=my-test-0, dir=/opt/middleware/kafka_2.11-1.1.0/logs] Loading producer state from offset 0 with message format version 2 (kafka.log.Log)
[2019-12-24 15:15:44,191] INFO [Log partition=my-test-0, dir=/opt/middleware/kafka_2.11-1.1.0/logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 44 ms (kafka.log.Log)
[2019-12-24 15:15:44,193] INFO Created log for partition my-test-0 in /opt/middleware/kafka_2.11-1.1.0/logs with properties {compression.type -> producer, message.format.version -> 1.1-IV0, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2019-12-24 15:15:44,194] INFO [Partition my-test-0 broker=0] No checkpointed highwatermark is found for partition my-test-0 (kafka.cluster.Partition)
[2019-12-24 15:15:44,196] INFO Replica loaded for partition my-test-0 with initial high watermark 0 (kafka.cluster.Replica)
[2019-12-24 15:15:44,221] INFO [Partition my-test-0 broker=0] my-test-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
[2019-12-24 15:15:44,250] INFO [ReplicaAlterLogDirsManager on broker 0] Added fetcher for partitions List() (kafka.server.ReplicaAlterLogDirsManager)

9.2 查看Topic

bin/kafka-topics.sh --describe --zookeeper 127.0.0.1:2181

[root@instance-n4r06itt kafka_2.11-1.1.0]# bin/kafka-topics.sh --describe --zookeeper 127.0.0.1:2181
Topic:my-test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: my-test	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

9.3 创建生产者

bin/kafka-console-producer.sh --broker-list 192.168.16.4:9092 --topic my-test

在这里插入图片描述

9.4 创建消费者

bin/kafka-console-consumer.sh --bootstrap-server 192.168.16.4:9092 --from-beginning --topic my-test

在这里插入图片描述9.5 遇到的问题。
kafka生产者、消费者，生产的数据必须为server.properties里面的listeners IP加PORT。不然不能正常消费。

至此，linux系统安装kafka成功。

谦奕爸爸
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
kafka-schema-registry-client-6.2.2.jar
04-25
mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=6.2.2 -Dfile=/root/kafka-schema-registry-client-6.2.2.jar -Dpackaging=jar 官网下载地址 packages....
Kafka——内外网访问的设置
我的博客
04-19 4211
外网访问kafka
kafka集群安装和使用
weixin_30879169的博客
08-17 94
kafka(1)kafka是一个分布式的消息缓存系统(2)kafka集群中的服务器都叫做broker(3)kafka有两类客户端,一个叫做producer(消息生产者),一类叫做consumer(消息消费者),客户端和broker服务器之间采用TCP协议连接(4)kafka中的消息可以通过topic进行区分,而且每一个消息topic都会被分区,以分担消息服务器的负载(5)每一个分区都可以...
部署kafka时遇到的坑：kafka.common.KafkaException: Socket server failed to bind to xx:9092: Cannot assign req
weixin_44943959的博客
03-02 9838
kafka学习
kafka环境搭建-启动失败：ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown
过山码农的博客
09-29 1934
kafka环境搭建-启动失败：ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: Socket server failed to bind to 47.100.19.248:9092: Cannot assign requested address.
org.apache.kafka.common.KafkaException: Socket server failed to bind to 114.115.20.100:9092: Cannot
热门推荐
风水道人
07-29 1万+
[2021-07-29 21:41:11,542] INFO Logs loading complete in 282 ms. (kafka.log.LogManager) [2021-07-29 21:41:11,552] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2021-07-29 21:41:11,553] INFO Starting log flusher with a defaul.
kafka-2.13-3.4.0.tgz
02-10
Kafka是一种高吞吐量的分布式发布订阅消息系统
kafka-eagle-bin-2.1.0.tar.gz
07-05
kafka-eagle-bin-2.1.0.tar.gz 2022年7月份下载，最新版
kafka-clients-2.4.1-API文档-中文版.zip
06-06
赠送jar包：kafka-clients-2.4.1.jar； 赠送原API文档：kafka-clients-2.4.1-javadoc.jar； 赠送源代码：kafka-clients-2.4.1-sources.jar； 赠送Maven依赖信息文件：kafka-clients-2.4.1.pom； 包含翻译后的API文档...
kafka-connect-oracle:Kafka Source Connector for Oracle
05-28
Kafka ConnectOraclekafka-connect-oracle是一个Kafka源连接器，用于从Oracle数据库捕获所有基于行的DML更改并将这些更改流式传输到Kafka。 变更数据捕获逻辑基于Oracle LogMiner解决方案。 仅从Oracle中提取已提交...
kafka使用时常见的几个错误汇总
paulwang040的博客
11-26 6649
kafka常见错误
【已解决】kafka使用遇到的坑
青苔小榭
08-06 1万+
文章目录1. [kafka使用文档](http://kafka.apache.org/quickstart)2. 在kafka启动zookeeper3. 运行kafka4. 再启动kafka 1. kafka使用文档 2. 在kafka启动zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties 遇到问题： [2020-08-06 11:22:05,915] ERROR Unexpected exception, exiting
手把手教你 在linux上安装kafka
向热爱自己一样，热爱这个世界。
03-17 5534
手把手教你 在linux上安装kafka
启动kafka报错
dengmanzhou3124的博客
07-28 2185
执行启动kafka服务的命令时报错：nohup bin/kafka-server-start.sh config/server.properties 2>&1 & [2019-07-28 12:58:44,760] ERROR [ReplicaManager broker=1] Error while making broker the follower for ...
linux安装kafka
永恩&猫咪的博客
03-25 3231
首先进入官网下载kafka 将下载的kafka压缩包 放入linux服务器的指定目录 解压kafka 命令 tar -xzf kafka_2.12-3.1.0.tgz 进入kafka目录命令 cd kafka_2.12-3.1.0/ 启动zookeeper命令 bin/zookeeper-server-start.sh config/zookeeper.properties 这个是 窗口模式，后台启动zookeeper命令 bin/zookeeper-server-start.sh config.
Linux下kafka的使用
记得微笑的博客
06-01 2176
1、启动 Zookeeper 服务，在 kafka 的根目录下使用命令 Kafka用到了Zookeeper，，先开启zookeeper，如下图所示： 下面用一个单实例的Zookkeeper服务，可以在命令结尾处加个&符号，这样就可以启动后离开控制台 bin/zookeeper-server-start.sh config/zookeeper.properties & 2、启动 kafka，在 kafka 的根目录下使用命令，如下图所示： bin/kafka-server-start.
Linux系统安装和使用Kafka教程。
蓝易云
06-20 1058
这只是一个简单的演示，展示了如何在Linux系统上安装和使用Kafka。实际使用中，你可能需要更复杂的配置和进一步的学习，以满足你的特定需求。你可以参考Kafka官方文档和相关教程，以获取更详细的指南和更多高级配置选项。安装和使用Kafka是一个相对复杂的过程，涉及多个步骤和配置。
Linux安装Kafka（包括单机版和集群版）
持续学习，坚持原创，分享技术笔记及经验。
08-21 829
Linux安装Kafka，包括单机版和集群版。
linux环境kafka安装及配置
W_StackOverFlow_W的博客
10-20 1万+
linux环境kafka安装及配置过程（含zookeeper）
kafka kafka-run-class kafka.tools.GetOffsetshell
最新发布
08-24
kafka-run-class是kafka提供的一个脚本工具，用于在命令行中执行指定的类。kafka.tools.GetOffsetShell是用于获取指定topic在指定时间点的偏移量信息的工具。 具体来说，kafka.tools.GetOffsetShell通过指定topic...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值