部署Kafka

已于 2023-12-20 15:41:58 修改
阅读量1.3k 收藏 1
点赞数
文章标签： kafka 分布式
于 2023-09-21 22:40:45 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wsdhla/article/details/133076481
版权

kafka：kafka_2.13-3.5.1

NOTE: Your local environment must have Java 8+ installed.

Apache Kafka can be started using ZooKeeper or KRaft. To get started with either configuration follow one the sections below but not both.

目录

1 Windows单机

1.1 Kafka with KRaft

1.1.1 Generate a Cluster UUID 

1.1.2 Format Log Directories

1.1.3 Start the Kafka Server

1.2 Kafka with ZooKeeper

1.2.1 Start the ZooKeeper service

1.2.2 Start the Kafka broker service

2 Windows集群（同一台物理机）

2.1 Kafka with KRaft

2.1.1 拷贝3份kafka应用

2.1.2 分别修改配置文件

2.1.3 Generate a Cluster UUID 

2.1.4 Format Log Directories

2.1.5 Start the Kafka Server

2.2 Kafka with ZooKeeper

2.2.1 同样需要3份kafka应用

2.2.2 分别修改配置文件

2.2.3 Start the ZooKeeper service 

2.2.4 Start the Kafka broker service

3 Linux单机

3.1 Kafka with KRaft

3.2 Kafka with ZooKeeper

4 Linux集群（3台虚拟机）

4.1 Kafka with KRaft

4.1 Kafka with ZooKeeper

常见异常

异常1：

异常2：

异常3：

异常4：

异常5：

异常6：

异常7：

异常8：

Error Codes

1 Windows单机

1.1 Kafka with KRaft

1.1.1 Generate a Cluster UUID 

%kafka_home%\bin\windows\kafka-storage.bat random-uuid

KAFKA_CLUSTER_ID：7W5iXjO2SESIaSv770eIyA

1.1.2 Format Log Directories

server.properties采用默认配置：

############################# Server Basics #############################

# The role of this server. Setting this puts us in KRaft mode
process.roles=broker,controller

# The node id associated with this instance's roles
node.id=1

# The connect string for the controller quorum
controller.quorum.voters=1@localhost:9093




############################# Socket Server Settings #############################

# The address the socket server listens on.
# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092,CONTROLLER://:9093

# Name of listener used for communication between brokers.
inter.broker.listener.name=PLAINTEXT

# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://localhost:9092

# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.names=CONTROLLER


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kraft-combined-logs
%kafka_home%\bin\windows\kafka-storage.bat format -t 7W5iXjO2SESIaSv770eIyA -c ../../config/kraft/server.properties

1.1.3 Start the Kafka Server

%kafka_home%\bin\windows\kafka-server-start.bat ../../config/kraft/server.properties

报错了

[2023-09-20 14:41:35,667] ERROR [SharedServer id=1] Got exception while starting SharedServer (kafka.server.SharedServer)
java.io.UncheckedIOException: Error while writing the Quorum status from the file C:\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state
        at org.apache.kafka.raft.FileBasedStateStore.writeElectionStateToFile(FileBasedStateStore.java:155)
        at org.apache.kafka.raft.FileBasedStateStore.writeElectionState(FileBasedStateStore.java:128)
        at org.apache.kafka.raft.QuorumState.transitionTo(QuorumState.java:477)
        at org.apache.kafka.raft.QuorumState.initialize(QuorumState.java:212)
        at org.apache.kafka.raft.KafkaRaftClient.initialize(KafkaRaftClient.java:370)
        at kafka.raft.KafkaRaftManager.buildRaftClient(RaftManager.scala:248)
        at kafka.raft.KafkaRaftManager.<init>(RaftManager.scala:174)
        at kafka.server.SharedServer.start(SharedServer.scala:247)
        at kafka.server.SharedServer.startForController(SharedServer.scala:129)
        at kafka.server.ControllerServer.startup(ControllerServer.scala:197)
        at kafka.server.KafkaRaftServer.$anonfun$startup$1(KafkaRaftServer.scala:95)
        at kafka.server.KafkaRaftServer.$anonfun$startup$1$adapted(KafkaRaftServer.scala:95)
        at scala.Option.foreach(Option.scala:437)
        at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:95)
        at kafka.Kafka$.main(Kafka.scala:113)
        at kafka.Kafka.main(Kafka.scala)
Caused by: java.nio.file.FileSystemException: C:\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state.tmp -> C:\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state: 另一个程序正在使用此文件，进程无法访问。

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:950)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:933)
        at org.apache.kafka.raft.FileBasedStateStore.writeElectionStateToFile(FileBasedStateStore.java:152)
        ... 15 more
        Suppressed: java.nio.file.FileSystemException: C:\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state.tmp -> C:\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state: 另一个程序正在使用此文件，进程无法访问。

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:947)
                ... 17 more
[2023-09-20 14:41:35,671] INFO [ControllerServer id=1] Waiting for controller quorum voters future (kafka.server.ControllerServer)
[2023-09-20 14:41:35,673] INFO [ControllerServer id=1] Finished waiting for controller quorum voters future (kafka.server.ControllerServer)
[2023-09-20 14:41:35,680] ERROR Encountered fatal fault: caught exception (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
java.lang.NullPointerException
        at kafka.server.ControllerServer.startup(ControllerServer.scala:210)
        at kafka.server.KafkaRaftServer.$anonfun$startup$1(KafkaRaftServer.scala:95)
        at kafka.server.KafkaRaftServer.$anonfun$startup$1$adapted(KafkaRaftServer.scala:95)
        at scala.Option.foreach(Option.scala:437)
        at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:95)
        at kafka.Kafka$.main(Kafka.scala:113)
        at kafka.Kafka.main(Kafka.scala)

各种搜索也无能为力，只能搜到问题，搜不到答案

在Windows上就这样草草了之了......我准备放弃了

1.2 Kafka with ZooKeeper

1.2.1 Start the ZooKeeper service

%kafka_home%\bin\windows\zookeeper-server-start.bat ../../config/zookeeper.properties

1.2.2 Start the Kafka broker service

%kafka_home%\bin\windows\kafka-server-start.bat ../../config/server.properties

2 Windows集群（同一台物理机）

2.1 Kafka with KRaft

2.1.1 拷贝3份kafka应用

2.1.2 分别修改配置文件

kafka_cluster_node1\config\kraft\server.properties

process.roles=broker,controller

node.id=1

controller.quorum.voters=1@localhost:9093,2@localhost:9093,3@localhost:9093

listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

advertised.listeners=PLAINTEXT://localhost:9092

log.dirs=/kafka_cluster_node1/logs/tmp/kraft-combined-logs

kafka_cluster_node2\config\kraft\server.properties

process.roles=broker,controller

node.id=2

controller.quorum.voters=1@localhost:9093,2@localhost:9093,3@localhost:9093

listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

advertised.listeners=PLAINTEXT://localhost:9092

log.dirs=/kafka_cluster_node2/logs/tmp/kraft-combined-logs

kafka_cluster_node3\config\kraft\server.properties

process.roles=broker,controller

node.id=3

controller.quorum.voters=1@localhost:9093,2@localhost:9093,3@localhost:9093

listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

advertised.listeners=PLAINTEXT://localhost:9092

log.dirs=/kafka_cluster_node3/logs/tmp/kraft-combined-logs

2.1.3 Generate a Cluster UUID 

所有的节点一定要使用统一的Cluster UUID，否则会报异常：

ERROR [RaftManager id=1] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

使用其中任意一个目录生成集群ID

kafka_cluster_node1\bin\windows\kafka-storage.bat random-uuid

2.1.4 Format Log Directories

分别初始化日志目录

kafka_cluster_node1\bin\kafka-storage.bat format -t IFxmE1eDTfSOX-U_ZV7ntw -c ../../config/kraft/server.properties

kafka_cluster_node2\bin\kafka-storage.bat format -t IFxmE1eDTfSOX-U_ZV7ntw -c ../../config/kraft/server.properties

kafka_cluster_node3\bin\kafka-storage.bat format -t IFxmE1eDTfSOX-U_ZV7ntw -c ../../config/kraft/server.properties

2.1.5 Start the Kafka Server

分别启动服务

kafka_cluster_node1\bin\kafka-server-start.bat ../../config/kraft/server.properties

kafka_cluster_node2\bin\kafka-server-start.bat ../../config/kraft/server.properties

kafka_cluster_node3\bin\kafka-server-start.bat ../../config/kraft/server.properties

依然报错！！！ 

kraft到底支不支持Windows啊？？？还是kafka版本的问题？？？

放弃！！！

2.2 Kafka with ZooKeeper

需要1个zk节点，3个kafka节点。

2.2.1 同样需要3份kafka应用

2.2.2 分别修改配置文件

kafka_cluster_node1\config\server.properties

broker.id=1

zookeeper.connect=localhost:2181

listeners=PLAINTEXT://:9092

advertised.listeners=PLAINTEXT://kafkaNode1:9092

log.dirs=/kafka_cluster_node1/logs/tmp/kafka-logs

 kafka_cluster_node2\config\server.properties

broker.id=2

zookeeper.connect=localhost:2181

listeners=PLAINTEXT://:9093

advertised.listeners=PLAINTEXT://kafkaNode2:9092

log.dirs=/kafka_cluster_node2/logs/tmp/kafka-logs

kafka_cluster_node3\config\server.properties

broker.id=3

zookeeper.connect=localhost:2181

listeners=PLAINTEXT://:9094

advertised.listeners=PLAINTEXT://kafkaNode3:9092

log.dirs=/kafka_cluster_node3/logs/tmp/kafka-logs

2.2.3 Start the ZooKeeper service 

可以启动独立部署的zk，也可以用kafka自带的zk

%kafka_home%\bin\windows\zookeeper-server-start.bat ../../config/zookeeper.properties

2.2.4 Start the Kafka broker service

分别启动kafka节点

kafka_cluster_node1\bin\windows\kafka-server-start.bat ../../config/server.properties

kafka_cluster_node2\bin\windows\kafka-server-start.bat ../../config/server.properties

kafka_cluster_node3\bin\windows\kafka-server-start.bat ../../config/server.properties

集群部署成功！！！

但是应用发送消息尚有问题，留个hook在此。

3 Linux单机

3.1 Kafka with KRaft

操作与windows一致，一把就成功，没有任何问题。（果然！！！）

3.2 Kafka with ZooKeeper

还没有验证。

4 Linux集群（3台虚拟机）

4.1 Kafka with KRaft

与Windows除了部分命令、目录不一样之外，其余操作步骤完全一样。

node1配置文件

[server.properties]

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@XXX.XXX.XXX.28:9093,2@XXX.XXX.XXX.29:9093,3@XXX.XXX.XXX.30:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://XXX.XXX.XXX.28:9092

node2配置文件

[server.properties]

process.roles=broker,controller
node.id=2
controller.quorum.voters=1@XXX.XXX.XXX.28:9093,2@XXX.XXX.XXX.29:9093,3@XXX.XXX.XXX.30:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://XXX.XXX.XXX.29:9092

node3配置文件

[server.properties]

process.roles=broker,controller
node.id=3
controller.quorum.voters=1@XXX.XXX.XXX.28:9093,2@XXX.XXX.XXX.29:9093,3@XXX.XXX.XXX.30:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://XXX.XXX.XXX.30:9092

4.1 Kafka with ZooKeeper

常见异常

异常1：

这是因为单节点启动时产生的 tmp\kafka-logs\meta.properties文件中broker.id与现在集群的各节点不一致的原因（如果前面各节点输出的日志目录不同也不会产生这个问题）

#Thu Sep 21 16:56:50 CST 2023
cluster.id=bkuCLgUiRYe5CSnptF6ZVQ
version=0
broker.id=0

可以修改为一致或者直接删除。

异常2：

由于是在同一个机器上做集群，advertised.listeners配置需要区分，可以通过修改etc/hosts实现

localhost kafkaNode1
localhost kafkaNode2
localhost kafkaNode3

异常3：

还是因为在同一个主机上做集群的原因，listeners端口要区分开，可以分别设置为9092、9093、9094解决。 

异常4：

hosts文件中配置域名映射时要使用127.0.0.1，不能使用localhost。 

异常5：

应用端异常，说明spring.kafka.bootstrap-servers配置错误，应该配置advertised.listeners的地址。

异常6：

悬而未决

应用端发送消息时出现异常Error: NOT_LEADER_OR_FOLLOWER，

异常7：

linux集群环境

就是防火墙原因，三个节点9093端口不通，添加iptables规则，问题解决。

异常8：

客户端生产消息：

[Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

已解决：

除了要检查配置文件中的listeners和advertised.listeners之外，还得看下spring配置文件配置项是否正确：

错误：kafka.bootstrap-servers
正确：spring.kafka.bootstrap-servers

异常9：

zk模式下报错

Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition test-topic01-0 could be determined

配置已解决：

[server.properties]

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafkaNode0:9092

配置hosts映射

127.0.0.1 kafkaNode0

Error Codes

官方文档：Apache Kafka 

We use numeric codes to indicate what problem occurred on the server. These can be translated by the client into exceptions or whatever the appropriate error handling mechanism in the client language. Here is a table of the error codes currently in use:

ERRORCODERETRIABLEDESCRIPTION
UNKNOWN_SERVER_ERROR-1FalseThe server experienced an unexpected error when processing the request.
NONE0False
OFFSET_OUT_OF_RANGE1FalseThe requested offset is not within the range of offsets maintained by the server.
CORRUPT_MESSAGE2TrueThis message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted topic, or is otherwise corrupt.
UNKNOWN_TOPIC_OR_PARTITION3TrueThis server does not host this topic-partition.
INVALID_FETCH_SIZE4FalseThe requested fetch size is invalid.
LEADER_NOT_AVAILABLE5TrueThere is no leader for this topic-partition as we are in the middle of a leadership election.
NOT_LEADER_OR_FOLLOWER6TrueFor requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.
REQUEST_TIMED_OUT7TrueThe request timed out.
BROKER_NOT_AVAILABLE8FalseThe broker is not available.
REPLICA_NOT_AVAILABLE9TrueThe replica is not available for the requested topic-partition. Produce/Fetch requests and other requests intended only for the leader or follower return NOT_LEADER_OR_FOLLOWER if the broker is not a replica of the topic-partition.
MESSAGE_TOO_LARGE10FalseThe request included a message larger than the max message size the server will accept.
STALE_CONTROLLER_EPOCH11FalseThe controller moved to another broker.
OFFSET_METADATA_TOO_LARGE12FalseThe metadata field of the offset request was too large.
NETWORK_EXCEPTION13TrueThe server disconnected before a response was received.
COORDINATOR_LOAD_IN_PROGRESS14TrueThe coordinator is loading and hence can't process requests.
COORDINATOR_NOT_AVAILABLE15TrueThe coordinator is not available.
NOT_COORDINATOR16TrueThis is not the correct coordinator.
INVALID_TOPIC_EXCEPTION17FalseThe request attempted to perform an operation on an invalid topic.
RECORD_LIST_TOO_LARGE18FalseThe request included message batch larger than the configured segment size on the server.
NOT_ENOUGH_REPLICAS19TrueMessages are rejected since there are fewer in-sync replicas than required.
NOT_ENOUGH_REPLICAS_AFTER_APPEND20TrueMessages are written to the log, but to fewer in-sync replicas than required.
INVALID_REQUIRED_ACKS21FalseProduce request specified an invalid value for required acks.
ILLEGAL_GENERATION22FalseSpecified group generation id is not valid.
INCONSISTENT_GROUP_PROTOCOL23FalseThe group member's supported protocols are incompatible with those of existing members or first group member tried to join with empty protocol type or empty protocol list.
INVALID_GROUP_ID24FalseThe configured groupId is invalid.
UNKNOWN_MEMBER_ID25FalseThe coordinator is not aware of this member.
INVALID_SESSION_TIMEOUT26FalseThe session timeout is not within the range allowed by the broker (as configured by group.min.session.timeout.ms and group.max.session.timeout.ms).
REBALANCE_IN_PROGRESS27FalseThe group is rebalancing, so a rejoin is needed.
INVALID_COMMIT_OFFSET_SIZE28FalseThe committing offset data size is not valid.
TOPIC_AUTHORIZATION_FAILED29FalseTopic authorization failed.
GROUP_AUTHORIZATION_FAILED30FalseGroup authorization failed.
CLUSTER_AUTHORIZATION_FAILED31FalseCluster authorization failed.
INVALID_TIMESTAMP32FalseThe timestamp of the message is out of acceptable range.
UNSUPPORTED_SASL_MECHANISM33FalseThe broker does not support the requested SASL mechanism.
ILLEGAL_SASL_STATE34FalseRequest is not valid given the current SASL state.
UNSUPPORTED_VERSION35FalseThe version of API is not supported.
TOPIC_ALREADY_EXISTS36FalseTopic with this name already exists.
INVALID_PARTITIONS37FalseNumber of partitions is below 1.
INVALID_REPLICATION_FACTOR38FalseReplication factor is below 1 or larger than the number of available brokers.
INVALID_REPLICA_ASSIGNMENT39FalseReplica assignment is invalid.
INVALID_CONFIG40FalseConfiguration is invalid.
NOT_CONTROLLER41TrueThis is not the correct controller for this cluster.
INVALID_REQUEST42FalseThis most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.
UNSUPPORTED_FOR_MESSAGE_FORMAT43FalseThe message format version on the broker does not support the request.
POLICY_VIOLATION44FalseRequest parameters do not satisfy the configured policy.
OUT_OF_ORDER_SEQUENCE_NUMBER45FalseThe broker received an out of order sequence number.
DUPLICATE_SEQUENCE_NUMBER46FalseThe broker received a duplicate sequence number.
INVALID_PRODUCER_EPOCH47FalseProducer attempted to produce with an old epoch.
INVALID_TXN_STATE48FalseThe producer attempted a transactional operation in an invalid state.
INVALID_PRODUCER_ID_MAPPING49FalseThe producer attempted to use a producer id which is not currently assigned to its transactional id.
INVALID_TRANSACTION_TIMEOUT50FalseThe transaction timeout is larger than the maximum value allowed by the broker (as configured by transaction.max.timeout.ms).
CONCURRENT_TRANSACTIONS51TrueThe producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing.
TRANSACTION_COORDINATOR_FENCED52FalseIndicates that the transaction coordinator sending a WriteTxnMarker is no longer the current coordinator for a given producer.
TRANSACTIONAL_ID_AUTHORIZATION_FAILED53FalseTransactional Id authorization failed.
SECURITY_DISABLED54FalseSecurity features are disabled.
OPERATION_NOT_ATTEMPTED55FalseThe broker did not attempt to execute this operation. This may happen for batched RPCs where some operations in the batch failed, causing the broker to respond without trying the rest.
KAFKA_STORAGE_ERROR56TrueDisk error when trying to access log file on the disk.
LOG_DIR_NOT_FOUND57FalseThe user-specified log directory is not found in the broker config.
SASL_AUTHENTICATION_FAILED58FalseSASL Authentication failed.
UNKNOWN_PRODUCER_ID59FalseThis exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.
REASSIGNMENT_IN_PROGRESS60FalseA partition reassignment is in progress.
DELEGATION_TOKEN_AUTH_DISABLED61FalseDelegation Token feature is not enabled.
DELEGATION_TOKEN_NOT_FOUND62FalseDelegation Token is not found on server.
DELEGATION_TOKEN_OWNER_MISMATCH63FalseSpecified Principal is not valid Owner/Renewer.
DELEGATION_TOKEN_REQUEST_NOT_ALLOWED64FalseDelegation Token requests are not allowed on PLAINTEXT/1-way SSL channels and on delegation token authenticated channels.
DELEGATION_TOKEN_AUTHORIZATION_FAILED65FalseDelegation Token authorization failed.
DELEGATION_TOKEN_EXPIRED66FalseDelegation Token is expired.
INVALID_PRINCIPAL_TYPE67FalseSupplied principalType is not supported.
NON_EMPTY_GROUP68FalseThe group is not empty.
GROUP_ID_NOT_FOUND69FalseThe group id does not exist.
FETCH_SESSION_ID_NOT_FOUND70TrueThe fetch session ID was not found.
INVALID_FETCH_SESSION_EPOCH71TrueThe fetch session epoch is invalid.
LISTENER_NOT_FOUND72TrueThere is no listener on the leader broker that matches the listener on which metadata request was processed.
TOPIC_DELETION_DISABLED73FalseTopic deletion is disabled.
FENCED_LEADER_EPOCH74TrueThe leader epoch in the request is older than the epoch on the broker.
UNKNOWN_LEADER_EPOCH75TrueThe leader epoch in the request is newer than the epoch on the broker.
UNSUPPORTED_COMPRESSION_TYPE76FalseThe requesting client does not support the compression type of given partition.
STALE_BROKER_EPOCH77FalseBroker epoch has changed.
OFFSET_NOT_AVAILABLE78TrueThe leader high watermark has not caught up from a recent leader election so the offsets cannot be guaranteed to be monotonically increasing.
MEMBER_ID_REQUIRED79FalseThe group member needs to have a valid member id before actually entering a consumer group.
PREFERRED_LEADER_NOT_AVAILABLE80TrueThe preferred leader was not available.
GROUP_MAX_SIZE_REACHED81FalseThe consumer group has reached its max size.
FENCED_INSTANCE_ID82FalseThe broker rejected this static consumer since another consumer with the same group.instance.id has registered with a different member.id.
ELIGIBLE_LEADERS_NOT_AVAILABLE83TrueEligible topic partition leaders are not available.
ELECTION_NOT_NEEDED84TrueLeader election not needed for topic partition.
NO_REASSIGNMENT_IN_PROGRESS85FalseNo partition reassignment is in progress.
GROUP_SUBSCRIBED_TO_TOPIC86FalseDeleting offsets of a topic is forbidden while the consumer group is actively subscribed to it.
INVALID_RECORD87FalseThis record has failed the validation on broker and hence will be rejected.
UNSTABLE_OFFSET_COMMIT88TrueThere are unstable offsets that need to be cleared.
THROTTLING_QUOTA_EXCEEDED89TrueThe throttling quota has been exceeded.
PRODUCER_FENCED90FalseThere is a newer producer with the same transactionalId which fences the current one.
RESOURCE_NOT_FOUND91FalseA request illegally referred to a resource that does not exist.
DUPLICATE_RESOURCE92FalseA request illegally referred to the same resource twice.
UNACCEPTABLE_CREDENTIAL93FalseRequested credential would not meet criteria for acceptability.
INCONSISTENT_VOTER_SET94FalseIndicates that the either the sender or recipient of a voter-only request is not one of the expected voters
INVALID_UPDATE_VERSION95FalseThe given update version was invalid.
FEATURE_UPDATE_FAILED96FalseUnable to update finalized features due to an unexpected server error.
PRINCIPAL_DESERIALIZATION_FAILURE97FalseRequest principal deserialization failed during forwarding. This indicates an internal error on the broker cluster security setup.
SNAPSHOT_NOT_FOUND98FalseRequested snapshot was not found
POSITION_OUT_OF_RANGE99FalseRequested position is not greater than or equal to zero, and less than the size of the snapshot.
UNKNOWN_TOPIC_ID100TrueThis server does not host this topic ID.
DUPLICATE_BROKER_REGISTRATION101FalseThis broker ID is already in use.
BROKER_ID_NOT_REGISTERED102FalseThe given broker ID was not registered.
INCONSISTENT_TOPIC_ID103TrueThe log's topic ID did not match the topic ID in the request
INCONSISTENT_CLUSTER_ID104FalseThe clusterId in the request does not match that found on the server
TRANSACTIONAL_ID_NOT_FOUND105FalseThe transactionalId could not be found
FETCH_SESSION_TOPIC_ID_ERROR106TrueThe fetch session encountered inconsistent topic ID usage
INELIGIBLE_REPLICA107FalseThe new ISR contains at least one ineligible replica.
NEW_LEADER_ELECTED108FalseThe AlterPartition request successfully updated the partition state but the leader has changed.
OFFSET_MOVED_TO_TIERED_STORAGE109FalseThe requested offset is moved to tiered storage.
FENCED_MEMBER_EPOCH110FalseThe member epoch is fenced by the group coordinator. The member must abandon all its partitions and rejoin.
UNRELEASED_INSTANCE_ID111FalseThe instance ID is still used by another member in the consumer group. That member must leave first.
UNSUPPORTED_ASSIGNOR112FalseThe assignor or its version range is not supported by the consumer group.
wsdhla
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
LinuxLinux部署kafka集群
03-01
【Linux】Linux部署kafka集群 包含：kafka，zookeeper，jdk压缩包
kafka安装部署
m0_46192647的博客
03-11 4万+
一、安装kafka 下载kafka两种方式 （1）、手动下载 下载地址：Apache Kafka （2）、也可以用命令下载（直接下载到服务器哦！）： 将安装包下载到/opt/monitor/kafka文件夹下面 下载命令： wget http://mirrors.hust.edu.cn/apache/kafka/2.0.0/kafka_2.12-2.0.0.tgz 手动下载移动到/opt/monitor/kafka目录下也可以 二、解压kafka tar -xzvf kafk
Kafka集群部署详解
Wu 小杰的博客
06-16 1219
1）kafka定义Kafka 被称为下一代分布式消息系统，由 Scala 和 Java编写，是非营利性组织ASF(Apache Software Foundation)基金会中的一个开源项目，比如:HTTP Server、Tomcat、Hadoop、ActiveMQ等开源软件都属于 Apache基金会的开源软件，类似的消息系统还有RabbitMQ、ActiveMQ、ZeroMQ。Kafka用于构建实时数据管道和流应用程序。它具有水平可伸缩性，容错性，快速性，可在数千家组织中同时投入生产协同工作。
Kafka安装与部署
weixin_44296846的博客
02-19 7123
大数据相关知识点 1. Kafka介绍 ​ Kafka是一种高吞吐量的分布式发布订阅消息系统，它可以处理消费者规模的网站中的所有动作流数据，具有高性能、持久化、多副本备份、横向扩展能力…… 分布式系统，易于向外扩展； 同时为发布和订阅提供高吞吐量; 支持多订阅者，当失败时能自动平衡消费者； 将消息持久化到磁盘，可用于批量消费； [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-hBSMojUf-1645255556442)(C:\Users\luffy\Desktop\lu
Kafka 概述
weixin_55609829的博客
07-21 339
Kafka 概述消息列队（MQ)为什么需要消息队列（MQ）使用消息队列的好处消息队列的两种模式kafka简介Kafka 定义Kafka 简介Kafka 的特性kafka系统架构 消息列队（MQ) 为什么需要消息队列（MQ） 主要原因是由于在高并发环境下，同步请求来不及处理，请求往往会发生阻塞。比如大量的请求并发访问数据库，导致行锁表锁，最后请求线程会堆积过多，从而触发 too many connection 错误，引发雪崩效应。 我们使用消息队列，通过异步处理请求，从而缓解系统的压力。消息队列常应用于异步处
Kafka 安装部署、集群启动、命令行操作 与 可视化工具 Kafka Tool
蚩尤后裔-汪茂雄
10-11 1万+
目录 Kafka 消息队列概述 Kafka 消息队列架构 Kafka 消息队列概述 1、Apache Kafka 是一个开源分布式消息队列/系统，该项目的目标是为处理实时数据提供一个统一、高通量、低等待的平台 2、Kafka 保存消息时根据 Topic（主题） 进行归类，发送消息者称为 Producer（生产者），消息接受者称为 Consumer（消费者），kafka 集群由多个 kafka 实例组成，每个实例(server) 成为 broker（经纪人）。 3、无论是 kafka 集群，还是
kafka启动异常InconsistentClusterIdException
咸花生的博客
05-03 941
异常信息 [2021-05-03 19:14:28,840] INFO Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Features{}, epoch=0). (kafka.server.FinalizedFeatureCache) [2021-05-03 19:14:28,855] INFO Cluster ID = I9TxciNjQiKV48quVZP2yw (kafka.
Docker部署Kafka以及Spring Kafka实现
01-11
这篇文章主要介绍了Docker部署Kafka以及Spring Kafka实现,文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下 从https://hub.docker.com/查找kafka 第三个活跃并...
利用docker和docker-compose部署单机kafka
01-20
Kafka需要Zookeeper来存储集群元数据，因此在部署Kafka之前通常需要先部署Zookeeper。 4. **Docker部署Kafka** 使用Docker部署Kafka非常便捷，只需两步： - 部署Zookeeper容器： ```shell docker run -d --name...
docker部署kafka单节点及springboot集成
最新发布
07-05
使用docker容器化部署kafka，zookeeper，并且在springboot中集成kafka完整示例工程代码 环境： 操作系统：win10 Docker：Docker Desktop 4.21.1 (114176)、Docker Engine v24.0.2 SpringBoot：2.7.15
Kafka集群部署
热门推荐
aiXueLing专栏
03-03 5万+
一、前提条件 1、部署Kafka集群搭建需要服务器至少3台，奇数台 2、Kafka的安装需要java环境，jdk1.8 3、Kafka安装包版本：kafka_2.11-0.10.0.1.tar.gz 4、假设3台服务器分别为：kafka1、kafka2、kafka3 kafka服务器名 IP 域名 kafka1 ...
Kafka（二）- Kafka集群部署
及时当勉励，岁月不待人
11-29 6328
Kafka（二）- Kafka集群部署
Kafka启动失败报错InconsistentClusterIdException
Mensonge的博客
10-30 1067
报错： 解决 ：切换到配置文件server.properties中存放kafka日志的目录，修改meta.properties文件中的clusterId为日志文件中新的 Cluster Id。重新启动。
kafka部署
YUEAwb的博客
04-12 1090
可见上篇文章，接着做就行（部署所有集群服务器）
kafka内外网访问配置问题及解决
like_java_的博客
08-18 2549
1.kafka在远程服务器安装，成功启动，正常测试消息发布，消费。 配置： listeners=PLAINTEXT://localhost:9092 zookeeper.connect=localhost:2181 2.使用springboot进行kafka整合，此时使用外网ip连接，报错： Connection to node 0 could not be established. Broker may not be available. 3.修改kafka的配置文件 #liste
NodeJS中使用Socket.IO
weixin_51072684的博客
03-28 2352
Socket.IO 是一个库，可以在客户端和服务器之间实现低延迟双向和基于事件的通信。它建立在WebSocket协议之上，并提供额外的保证，例如回退到 HTTP 长轮询或自动重新连接。
Kafka部署
weixin_50344742的博客
03-15 4246
一、环境准备 Kafka集群环境搭建，需要准备好一个zookeeper环境（集群），zookeeper需要java环境 tar xf /java_v1.8.0-201_zhuizhi_20200423 /usr/local/java.tar.gz ln -s /data/java/java_v1.8.0-201_zhuizhi_20200423 /usr/local/java cat /etc/profile.d/environment.sh #Java export JAVA_HOME=/usr/
Hello Kafka（六）——Kafka集群部署
天山老妖
02-24 1812
一、Kafka集群部署方案规划 1、操作系统选择 通常，生产环境应该将Kafka集群部署在Linux操作系统上，原因如下： （1）Kafka客户端底层使用了Java的selector，selector在Linux上的实现机制是epoll，而在Windows平台上的实现机制是select，因此Kafka部署在Linux上能够获得更高效的I/O性能。 （2）网络传输效率的差别。Kafka需要在磁盘和网络间进行大量数据传输，在Linux部署Kafka能够享受到零拷贝（Zero Copy）技术所带来的快速数
大数据技术之Kafka——Kafka的安装与部署
five小点心的博客
04-13 2043
（2）也可以用命令下载（直接下载到服务器哦！）：将安装包下载到/opt/monitor/kafka文件夹下面下载命令：手动下载移动到/opt/soft/kafka目录下也可以。
helm 部署kafka
09-01
使用Helm部署Kafka可以简化部署和管理过程。您可以使用以下命令来部署Kafka： ```shell helm install --name kafka --namespace kafka -f kafka-values.yaml bitnami/kafka ``` 这个命令将使用Helm安装Bitnami的Kafka chart。要成功执行此命令，请确保您已经添加了Bitnami的Helm仓库，并且已经在您的环境中配置了正确的命名空间和值文件。您可以使用以下命令来添加Bitnami的Helm仓库： ```shell helm repo add bitnami https://charts.bitnami.com/bitnami ``` 通过这个命令，您可以将Bitnami的Helm仓库添加到您的环境中。然后，您可以使用以下命令来查看已添加的仓库列表： ```shell helm repo list ``` 这个命令将显示您添加的所有Helm仓库的列表。 请注意，这只是部署Kafka的基本示例。根据您的需求，您可能需要调整部署配置和参数。我希望这个答案对您有帮助！<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* *2* [Helm 安装Kafka](https://blog.csdn.net/weixin_30471561/article/details/101474739)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] - *3* [Helm方式部署 zookeeper+kafka 集群 ——2023.05](https://blog.csdn.net/weixin_64124795/article/details/130772120)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] [ .reference_list ]

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

wsdhla

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值