Zookeeper的一次迁移故障

最新推荐文章于 2024-06-19 15:18:44 发布

五柳-先生

最新推荐文章于 2024-06-19 15:18:44 发布

阅读量1.3w

点赞数

分类专栏：分布式服务-zooKeeper

分布式服务-zooKeeper 专栏收录该内容

29 篇文章 0 订阅

订阅专栏

前阶段同事迁移Zookeeper（是给Kafka使用的以及flume使用）后发现所有Flume-producer/consumer端集体报错：

 
   
        07  
        Jan  
        2014  
        01 
        : 
        19 
        : 
        32 
        , 
        571  
        INFO  [conf-file-poller- 
        0 
        -SendThread(xxx: 
        2181 
        )] (org.apache.zookeeper.ClientCnxn$SendThread.startConnect: 
        1058 
        )  - Opening socket connection to server xxx: 
        2181 
       
 
        07  
        Jan  
        2014  
        01 
        : 
        19 
        : 
        32 
        , 
        572  
        INFO  [conf-file-poller- 
        0 
        -SendThread(xxx: 
        2181 
        )] (org.apache.zookeeper.ClientCnxn$SendThread.primeConnection: 
        947 
        )  - Socket connection established to xxx: 
        2181 
        , initiating session 
       
 
        07  
        Jan  
        2014  
        01 
        : 
        19 
        : 
        32 
        , 
        573  
        INFO  [conf-file-poller- 
        0 
        -SendThread(xxx: 
        2181 
        )] (org.apache.zookeeper.ClientCnxn$SendThread.run: 
        1183 
        )  - Unable to read additional data from server sessionid  
        0x142f42b91871911 
        , likely server has closed socket, closing socket connection and attempting reconnect 
       
 
        07  
        Jan  
        2014  
        01 
        : 
        19 
        : 
        32 
        , 
        845  
        INFO  [conf-file-poller- 
        0 
        -SendThread(xxx: 
        2181 
        )] (org.apache.zookeeper.ClientCnxn$SendThread.startConnect: 
        1058 
        )  - Opening socket connection to server xxx: 
        2181 
       
 
 

一直在不断的重试连接失败再重试，问同事说：网路连通性早就验证过，然后查看server端日志发现：

 
   
        2014 
        - 
        01 
        - 
        06  
        23 
        : 
        59 
        : 
        59 
        , 
        987  
        [myid: 
        1 
        ] - INFO  [NIOServerCxn.Factory: 
        0.0 
        . 
        0.0 
        / 
        0.0 
        . 
        0.0 
        : 
        2181 
        :NIOServerCnxnFactory 
        @197 
        ] - Accepted socket connection from /xxx: 
        45282 
       
 
        2014 
        - 
        01 
        - 
        06  
        23 
        : 
        59 
        : 
        59 
        , 
        987  
        [myid: 
        1 
        ] - WARN  [NIOServerCxn.Factory: 
        0.0 
        . 
        0.0 
        / 
        0.0 
        . 
        0.0 
        : 
        2181 
        :ZooKeeperServer 
        @793 
        ] - Connection request from old client xxx: 
        45282 
        ; will 
       
 
        be dropped  
        if  
        server is in r-o mode 
       
 
        2014 
        - 
        01 
        - 
        06  
        23 
        : 
        59 
        : 
        59 
        , 
        987  
        [myid: 
        1 
        ] - INFO  [NIOServerCxn.Factory: 
        0.0 
        . 
        0.0 
        / 
        0.0 
        . 
        0.0 
        : 
        2181 
        :ZooKeeperServer 
        @812 
        ] - Refusing session request  
        for  
        client xxx: 
        45282  
        as it 
       
 
        has seen zxid  
        0x60fd15564  
        our last zxid is  
        0x10000000f  
        client must  
        try  
        another server 
       
 
        2014 
        - 
        01 
        - 
        06  
        23 
        : 
        59 
        : 
        59 
        , 
        987  
        [myid: 
        1 
        ] - INFO  [NIOServerCxn.Factory: 
        0.0 
        . 
        0.0 
        / 
        0.0 
        . 
        0.0 
        : 
        2181 
        :NIOServerCnxn 
        @1001 
        ] - Closed socket connection  
        for  
        client xxx: 
        45282  
        (no se 
       
 
        ssion established  
        for  
        client) 
       
 
        2014 
        - 
        01 
        - 
        06  
        23 
        : 
        59 
        : 
        59 
        , 
        989  
        [myid: 
        1 
        ] - INFO  [NIOServerCxn.Factory: 
        0.0 
        . 
        0.0 
        / 
        0.0 
        . 
        0.0 
        : 
        2181 
        :NIOServerCnxnFactory 
        @197 
        ] - Accepted socket connection from xxx: 
        45285 
       
 
 

发现Flume还是保留原来的zxid，但是现在的zxid竟然是0，所以抛出异常！

 
        if  
        (connReq.getLastZxidSeen() > zkDb.dataTree.lastProcessedZxid) { 
       
        String msg =  
        "Refusing session request for client " 
       
        + cnxn.getRemoteSocketAddress() 
       
        +  
        " as it has seen zxid 0x" 
       
        + Long.toHexString(connReq.getLastZxidSeen()) 
       
        +  
        " our last zxid is 0x" 
       
        + Long.toHexString(getZKDatabase().getDataTreeLastProcessedZxid()) 
       
        +  
        " client must try another server" 
        ; 
       
        LOG.info(msg); 
       
        throw  
        new  
        CloseRequestException(msg); 
       
        }