shell更新kafka的offset到最新的位置

docker更新kafka的offset到最新的位置

1.进入docker容器，用来执行kafka相关的更新offset的命令；

docker ps | grep kafka | awk '{print $1}'

2.执行如下命令，用来将执行kafka集群中的groupId的offset更新为最新；

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.19.32.116:9192  --group fang-groupid-105  --reset-offsets --all-topics --to-latest --execute

其中，命令解读

• –group 表示需要更新offset的位置；
• –reset-offsets 将该groupId相关的offset重新为最新；
• –all-topics 表示该groupId相关联的所有topic全部执行
• –to-latest 表示执行更新offset到最新的位置

备注：https://blog.csdn.net/zzti_erlie/article/details/93637932

可能的执行结果如下：

bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.19.32.116:9192  --group fang-groupid-105  --reset-offsets --all-topics --to-latest --execute
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 29998; nested exception is: 
	java.net.BindException: Address in use (Bind failed)
sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 29998; nested exception is: 
	java.net.BindException: Address in use (Bind failed)
	at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:480)
	at sun.management.Agent.startAgent(Agent.java:262)
	at sun.management.Agent.startAgent(Agent.java:452)
Caused by: java.rmi.server.ExportException: Port already in use: 29998; nested exception is: 
	java.net.BindException: Address in use (Bind failed)
	at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
	at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
	at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
	at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
	at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:236)
	at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:213)
	at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:173)
	at sun.management.jmxremote.SingleEntryRegistry.<init>(SingleEntryRegistry.java:49)
	at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
	at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
	... 2 more
Caused by: java.net.BindException: Address in use (Bind failed)
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
	at java.net.ServerSocket.bind(ServerSocket.java:375)
	at java.net.ServerSocket.<init>(ServerSocket.java:237)
	at java.net.ServerSocket.<init>(ServerSocket.java:128)
	at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:45)
	at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:345)
	at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
	at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
	... 11 more

该错误参考文献：https://github.com/apache/kafka/pull/1983/commits/2c5d40e946bcc149b1a9b2c01eced4ae47a734c5

如需要修复该问题，参考网上的方式，加入的位置详见上文的链接；

需要修改/opt/kafka/bin/kafka-run-class.sh，即kafka-run-class.sh文件中部分内容；

################# 第一处添加如下：by fang ################################ 
ISKAFKASERVER="false"                                                              
if [[ "$*" =~ "kafka.Kafka" ]]; then                                               
    ISKAFKASERVER="true"                                                           
fi  

################## 第二处修改如下：by fang ###############################
# JMX port to use    
if [  $JMX_PORT ] && [ -z "$ISKAFKASERVER" ]; then          
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "                                             
fi 

备注：目前已经在172.19.32.101上已经修改了上面的配置，如果重启容器仍然需要重新配置该过程。即，对应的更新offset脚本可以在这个机器上面执行即可；

最后，重新执行操作即可，如下所示，示例topic：fang-topic-105已经将所有的offset置为最新；

bash-4.4# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.19.32.116:9192  --group fang-groupid-105  --reset-offsets --all-topics --to-latest --execute

TOPIC                          PARTITION  NEW-OFFSET     
fang-topic-105                 3          410            
fang-topic-105                 1          402            
fang-topic-105                 2          381            
fang-topic-105                 0          351            
fang-topic-105                 4          366            
fang-topic-105                 5          404      

如果需要批量的跟新groupid的offset值，使用如下脚本即可

#/bin/bash

echo "######################################"
echo "[note]: this batch will be update kafka offset to latest!!"
KAFKA_CONTAINER_ID=$(docker ps | grep kafka | awk '{print $1}' )
echo "KAFKA_CONTAINER_ID"=$KAFKA_CONTAINER_ID
echo "======================================"

docker exec -it $KAFKA_CONTAINER_ID /bin/bash -c "/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.19.32.116:9192  --group fang-groupid-105  --reset-offsets --all-topics --to-latest --execute; \
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.19.32.116:9192  --group fang-groupid-106  --reset-offsets --all-topics --to-latest --execute"

echo "[successfully] update kafka offset finished !!"

注意：当更新多个groupid的时候，注意加上英文分号; 如果入市需要换行显示加上反斜杠 \；