TX-LCN分布式事务框架源码解析（服务端-2-TxLcnInitializer初始化之TMRpcServer）

最新推荐文章于 2023-04-29 16:45:35 发布

jackson陈

最新推荐文章于 2023-04-29 16:45:35 发布

阅读量1.5k

点赞数 2

分类专栏：分布式事物 netty

本文链接：https://blog.csdn.net/cgj296645438/article/details/90639953

版权

分布式事物同时被 2 个专栏收录

11 篇文章 2 订阅

订阅专栏

netty

4 篇文章 0 订阅

订阅专栏

上篇文章我们通过注解@EnableTransactionManagerServer了解了服务端的启动初始化流程。最重要的TxLcnInitializer#init我们在本篇文章详细说明

先看下服务端实现了TxLcnInitializer接口的类有哪些

一共有五个实现类分别是

EnsureIdGenEngine类，用于处理和生成全局唯一id

MysqlLoggerHelper类，用于记录mysql的日志（不做细讲）

RpcNettyInitializer类，netty的初始化类

TMAutoCluster类，服务端的自动集群

TMRpcServer类，TM 服务端

其执行顺序是

1、TMRpcServer

这个类是RPC 服务端，TX-LCN框架底层采用的是netty进行服务端与客户端通信，TMRpcServer类的作用是启动netty服务端。

我们看init方法

public void init() {
        // 1. 配置
        if (rpcConfig.getWaitTime() <= 5) {
            //最大等待时间
            rpcConfig.setWaitTime(1000);
        }
        if (rpcConfig.getAttrDelayTime() < 0) {
            //延时时间 8s
            rpcConfig.setAttrDelayTime(txManagerConfig.getDtxTime());
        }

        // 2. 初始化RPC Server
        ManagerProperties managerProperties = new ManagerProperties();
        //心跳检测时间 5m
        managerProperties.setCheckTime(txManagerConfig.getHeartTime());
        //端口号
        managerProperties.setRpcPort(txManagerConfig.getPort());
        //ip
        managerProperties.setRpcHost(txManagerConfig.getHost());
        //委托NettyRpcServerInitializer类执行init方法
        rpcServerInitializer.init(managerProperties);
    }

这里主要配置了rpcconfig，设置了ManagerProperties的三个属性，这里ip需要说一下。ip是存在TxManagerConfig类中

其初始化ip是在构造方法中进行的代码如下

    public static final int PORT_CHANGE_VALUE = 100;

    @Autowired
    public TxManagerConfig(ServerProperties serverProperties) {
        this.port = Objects.requireNonNull(serverProperties.getPort(), "TM http port not configured?") +
                PORT_CHANGE_VALUE;
    }

如果工程的server.port=7900 ，则这里的port为7900+100 即8000，这里netty启动的端口是server.port+100。

1.1 NettyRpcServerInitializer的init方法如下

public void init(ManagerProperties managerProperties) {
        //为NettyContext设置类型与参数
        NettyContext.type = NettyType.server;
        NettyContext.params = managerProperties;
        
        nettyRpcServerChannelInitializer.setManagerProperties(managerProperties);

        int port = managerProperties.getRpcPort();
        //标准的netty服务端写法
        bossGroup = new NioEventLoopGroup();
        workerGroup = new NioEventLoopGroup();
        try {
            ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .option(ChannelOption.SO_BACKLOG, 100)
                    .handler(new LoggingHandler(LogLevel.INFO))
                    //配置channelhandler
                    .childHandler(nettyRpcServerChannelInitializer);

            // 启动服务.
            if (StringUtils.hasText(managerProperties.getRpcHost())) {
                b.bind(managerProperties.getRpcHost(), managerProperties.getRpcPort());
            } else {
                b.bind(port);
            }
            log.info("Socket started on {}:{} ",
                    StringUtils.hasText(managerProperties.getRpcHost()) ? managerProperties.getRpcHost() : "0.0.0.0", port);

        } catch (Exception e) {
            // Shut down all event loops to terminate all threads.
            e.printStackTrace();
        }
    }

设置的childHandler为NettyRpcServerChannelInitializer此类继承了ChannelInitializer会在客户端连接后去执行NettyRpcServerChannelInitializer的initChannel方法。

1.2 NettyRpcServerChannelInitializer的initChannel方法如下

protected void initChannel(Channel ch) throws Exception {
        //这两个类配合使用解决半包与粘包问题
        ch.pipeline().addLast(new LengthFieldPrepender(4, false));
        ch.pipeline().addLast(new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, 0, 4));
        //心跳
        ch.pipeline().addLast(new IdleStateHandler(managerProperties.getCheckTime(),
                managerProperties.getCheckTime(), managerProperties.getCheckTime(), TimeUnit.MILLISECONDS));
        
        //编码与解码
        ch.pipeline().addLast(new ObjectSerializerEncoder());
        ch.pipeline().addLast(new ObjectSerializerDecoder());

        //框架特有的Handler
        ch.pipeline().addLast(rpcCmdDecoder);
        ch.pipeline().addLast(new RpcCmdEncoder());
        ch.pipeline().addLast(socketManagerInitHandler);
        ch.pipeline().addLast(rpcAnswerHandler);
    }

半包见https://blog.csdn.net/cgj296645438/article/details/90667419

心跳见https://blog.csdn.net/cgj296645438/article/details/90902821

编码器ObjectSerializerEncoder继承MessageToByteEncoder（netty）进行编码，把对象转化成byte

解码器ObjectSerializerDecoder继承MessageToMessageDecoder（netty）进行二次解码把对象转换为对象

编码器与解码器使用的谷歌的protobuf工具。

框架handler

RpcCmdDecoder

RpcCmd是客户端服务端消息传输的载体的父类，即是消息。

下面看源码

public class RpcCmdDecoder extends SimpleChannelInboundHandler<NettyRpcCmd> {

    @Autowired
    private HeartbeatListener heartbeatListener;

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, NettyRpcCmd cmd) {
        String key = cmd.getKey();
        log.debug("cmd->{}", cmd);

        //心态数据包直接响应
        if (cmd.getMsg() != null && MessageConstants.ACTION_HEART_CHECK.equals(cmd.getMsg().getAction())) {
            if (NettyContext.currentType().equals(NettyType.client)) {
                //设置值
                heartbeatListener.onTcReceivedHeart(cmd);
                ctx.writeAndFlush(cmd);
                return;
            } else {
                heartbeatListener.onTmReceivedHeart(cmd);
                return;
            }
        }

        //需要响应的数据包
        if (!StringUtils.isEmpty(key)) {
            RpcContent rpcContent = cmd.loadRpcContent();
            if (rpcContent != null) {
                log.debug("got response message[Netty Handler]");
                rpcContent.setRes(cmd.getMsg());
                rpcContent.signal();
            } else {
                ctx.fireChannelRead(cmd);
            }
        } else {
            ctx.fireChannelRead(cmd);
        }
    }
}

从代码主要做了两件事

1、如果消息是心跳消息，判断当前是客户端还是服务端，如果是服务端则调用EnsureIdGenEngine类的onTmReceivedHeart方法去操作redis刷新机器号的超时时间。如果是客户端则调用TCSideRpcInitCallBack的onTcReceivedHeart方法为当前的消息设置MachineId

2、如果不是心跳消息，根据有无key值去做进一步处理，主要是设置rpccontent

RpcCmdEncoder

这个类很简单什么也没做只是把当前对象放入list里面，代码如下

public class RpcCmdEncoder extends MessageToMessageEncoder<NettyRpcCmd> {


    @Override
    protected void encode(ChannelHandlerContext ctx, NettyRpcCmd cmd, List<Object> out) throws Exception {
        log.debug("send->{}", cmd);
        out.add(cmd);
    }
}

SocketManagerInitHandler

代码如下

public class SocketManagerInitHandler extends ChannelInboundHandlerAdapter {

    private RpcCmd heartCmd;

    @Autowired
    private RpcConnectionListener rpcConnectionListener;

    public SocketManagerInitHandler() {
        MessageDto messageDto = new MessageDto();
        messageDto.setAction(MessageConstants.ACTION_HEART_CHECK);
        heartCmd = new NettyRpcCmd();
        heartCmd.setMsg(messageDto);
        heartCmd.setKey(RandomUtils.simpleKey());
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        super.channelActive(ctx);
        rpcConnectionListener.connect(ctx.channel().remoteAddress().toString());
        SocketManager.getInstance().addChannel(ctx.channel());
    }

    @Override
    public void channelInactive(ChannelHandlerContext ctx) throws Exception {
        super.channelInactive(ctx);
        String removeKey = ctx.channel().remoteAddress().toString();
        String appName = SocketManager.getInstance().getModuleName(removeKey);
        rpcConnectionListener.disconnect(removeKey,appName);
        SocketManager.getInstance().removeChannel(ctx.channel());
    }


    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
        //心跳配置
        if (IdleStateEvent.class.isAssignableFrom(evt.getClass())) {
            IdleStateEvent event = (IdleStateEvent) evt;
            if (event.state() == IdleState.READER_IDLE) {
                ctx.writeAndFlush(heartCmd);
            }
        }
    }

}

主要做了三件事

1、构造心跳消息

2、在有客户端连接上服务端是初始化SocketManager并把链接的channel保存到SocketManager里，在客户端断开链接是把channel从SocketManager上移除

3、和IdleStateHandler配合使用当IdleStateHandler一段时间没有监测到读或写请求，会执行此userEventTriggered方法向客户端发送心跳数据包，然后嘿嘿又和RpcCmdDecoder联合使用返回心跳数据包。

RpcAnswerHandler

应答handler

public class RpcAnswerHandler extends SimpleChannelInboundHandler<RpcCmd> {
    //服务端这里是ServerRpcAnswer
    @Autowired
    private RpcAnswer rpcClientAnswer;

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, RpcCmd cmd) {
        String remoteKey = ctx.channel().remoteAddress().toString();
        cmd.setRemoteKey(remoteKey);
        rpcClientAnswer.callback(cmd);
    }
}

ServerRpcAnswer代码如下

public class ServerRpcAnswer implements RpcAnswer, DisposableBean {

    private final RpcClient rpcClient;

    private final ExecutorService executorService;

    private final TxLcnManagerRpcBeanHelper rpcBeanHelper;

    @Autowired
    public ServerRpcAnswer(RpcClient rpcClient, TxLcnManagerRpcBeanHelper rpcBeanHelper, TxManagerConfig managerConfig) {
        managerConfig.setConcurrentLevel(
                Math.max(Runtime.getRuntime().availableProcessors() * 5, managerConfig.getConcurrentLevel()));
        this.rpcClient = rpcClient;
        this.executorService = Executors.newFixedThreadPool(managerConfig.getConcurrentLevel(),
                new ThreadFactoryBuilder().setDaemon(false).setNameFormat("tm-rpc-service-%d").build());
        this.rpcBeanHelper = rpcBeanHelper;
    }


    @Override
    public void callback(RpcCmd rpcCmd) {
        executorService.submit(() -> {
            try {
                TransactionCmd transactionCmd = parser(rpcCmd);
                String action = transactionCmd.getMsg().getAction();
                RpcExecuteService rpcExecuteService = rpcBeanHelper.loadManagerService(transactionCmd.getType());
                MessageDto messageDto = null;
                try {
                    Serializable message = rpcExecuteService.execute(transactionCmd);
                    messageDto = MessageCreator.okResponse(message, action);
                } catch (Throwable e) {
                    log.error("rpc execute service error. action: " + action, e);
                    messageDto = MessageCreator.failResponse(e, action);
                } finally {
                    // 对需要响应信息的请求做出响应
                    if (rpcCmd.getKey() != null) {
                        assert Objects.nonNull(messageDto);
                        try {
                            messageDto.setGroupId(rpcCmd.getMsg().getGroupId());
                            rpcCmd.setMsg(messageDto);
                            rpcClient.send(rpcCmd);
                        } catch (RpcException ignored) {
                        }
                    }
                }
            } catch (Throwable e) {
                if (rpcCmd.getKey() != null) {
                    log.info("send response.");
                    String action = rpcCmd.getMsg().getAction();
                    // 事务协调器业务未处理的异常响应服务器失败
                    rpcCmd.setMsg(MessageCreator.serverException(action));
                    try {
                        rpcClient.send(rpcCmd);
                        log.info("send response ok.");
                    } catch (RpcException ignored) {
                        log.error("requester:{} dead.", rpcCmd.getRemoteKey());
                    }
                }
            }
        });
    }

    @Override
    public void destroy() throws Exception {
        this.executorService.shutdown();
        this.executorService.awaitTermination(6, TimeUnit.SECONDS);
    }

    private TransactionCmd parser(RpcCmd rpcCmd) {
        TransactionCmd cmd = new TransactionCmd();
        cmd.setRequestKey(rpcCmd.getKey());
        cmd.setRemoteKey(rpcCmd.getRemoteKey());
        cmd.setType(LCNCmdType.parserCmd(rpcCmd.getMsg().getAction()));
        cmd.setGroupId(rpcCmd.getMsg().getGroupId());
        cmd.setMsg(rpcCmd.getMsg());
        return cmd;
    }

}

这个类是重点，所有的事物操作都在这里面，我们仔细分析

构造方法

    private final RpcClient rpcClient;

    private final ExecutorService executorService;

    private final TxLcnManagerRpcBeanHelper rpcBeanHelper;

    @Autowired
    public ServerRpcAnswer(RpcClient rpcClient, TxLcnManagerRpcBeanHelper rpcBeanHelper, TxManagerConfig managerConfig) {
        managerConfig.setConcurrentLevel(
                Math.max(Runtime.getRuntime().availableProcessors() * 5, managerConfig.getConcurrentLevel()));
        this.rpcClient = rpcClient;
        this.executorService = Executors.newFixedThreadPool(managerConfig.getConcurrentLevel(),
                new ThreadFactoryBuilder().setDaemon(false).setNameFormat("tm-rpc-service-%d").build());
        this.rpcBeanHelper = rpcBeanHelper;
    }

这里面三个参数

1、RpcClient，是用来通信的类，在服务端的实现是NettyRpcClient，部分代码如下可以看到都是和通信相关的

public class NettyRpcClient extends RpcClient {


    @Override
    public RpcResponseState send(RpcCmd rpcCmd) throws RpcException {
        return SocketManager.getInstance().send(rpcCmd.getRemoteKey(), rpcCmd);
    }


    @Override
    public RpcResponseState send(String remoteKey, MessageDto msg) throws RpcException {
        RpcCmd rpcCmd = new NettyRpcCmd();
        rpcCmd.setMsg(msg);
        rpcCmd.setRemoteKey(remoteKey);
        return send(rpcCmd);
    }

    @Override
    public MessageDto request(RpcCmd rpcCmd) throws RpcException {
        return request0(rpcCmd, -1);
    }

    private MessageDto request0(RpcCmd rpcCmd, long timeout) throws RpcException {
        if (rpcCmd.getKey() == null) {
            throw new RpcException("key must be not null.");
        }
        return SocketManager.getInstance().request(rpcCmd.getRemoteKey(), rpcCmd, timeout);
    }

  .............

但是实际通信的功能都委托给了SocketManager。

2、ExecutorService线程池不用多说，根据当前可用核心创建线程池

3、TxLcnManagerRpcBeanHelper获取spring中特定类型与姓名的bean的帮助类代码如下

public class TxLcnManagerRpcBeanHelper {


    /**
     * manager bean 名称格式
     * manager_%s_%s
     * manager:前缀 %s:业务处理(create,add,close)
     */
    private static final String RPC_BEAN_NAME_FORMAT = "rpc_%s";


    @Autowired
    private ApplicationContext spring;


    public String getServiceBeanName(LCNCmdType cmdType) {
        return String.format(RPC_BEAN_NAME_FORMAT, cmdType.getCode());
    }


    public RpcExecuteService loadManagerService(LCNCmdType cmdType) {
        return spring.getBean(getServiceBeanName(cmdType), RpcExecuteService.class);
    }

    public <T> T getByType(Class<T> type) {
        return spring.getBean(type);
    }
}

callback方法

public void callback(RpcCmd rpcCmd) {
        executorService.submit(() -> {
            try {
                TransactionCmd transactionCmd = parser(rpcCmd);
                //当前消息的操作类型比如创建事物组、加入事务组等
                String action = transactionCmd.getMsg().getAction();
                //根据操作类型获取具体的service
                RpcExecuteService rpcExecuteService = rpcBeanHelper.loadManagerService(transactionCmd.getType());
                MessageDto messageDto = null;
                try {
                    //执行execute
                    Serializable message = rpcExecuteService.execute(transactionCmd);
                    //如果正确执行封装响应消息
                    messageDto = MessageCreator.okResponse(message, action);
                } catch (Throwable e) {
                    log.error("rpc execute service error. action: " + action, e);
                    messageDto = MessageCreator.failResponse(e, action);
                } finally {
                    // 对需要响应信息的请求做出响应
                    if (rpcCmd.getKey() != null) {
                        assert Objects.nonNull(messageDto);
                        try {
                            messageDto.setGroupId(rpcCmd.getMsg().getGroupId());
                            rpcCmd.setMsg(messageDto);
                            //响应消息
                            rpcClient.send(rpcCmd);
                        } catch (RpcException ignored) {
                        }
                    }
                }
            } catch (Throwable e) {
                if (rpcCmd.getKey() != null) {
                    log.info("send response.");
                    String action = rpcCmd.getMsg().getAction();
                    // 事务协调器业务未处理的异常响应服务器失败
                    rpcCmd.setMsg(MessageCreator.serverException(action));
                    try {
                        rpcClient.send(rpcCmd);
                        log.info("send response ok.");
                    } catch (RpcException ignored) {
                        log.error("requester:{} dead.", rpcCmd.getRemoteKey());
                    }
                }
            }
        });
    }

主要是通过线程池进行提交任务操作，具体说明已经写到代码中。

TMRpcServer的内容还是挺多的，重点是几个handler的讲解。我们回顾下一个有这几个

LengthFieldPrepender 继承MessageToMessageEncoder 间接继承ChannelOutboundHandlerAdapter

LengthFieldBasedFrameDecoder 继承ByteToMessageDecoder 间接继承ChannelInboundHandlerAdapter

IdleStateHandler 继承ChannelDuplexHandler 间接继承ChannelInboundHandlerAdapter 间接实现ChannelOutboundHandler

ObjectSerializerEncoder 继承MessageToByteEncoder 间接继承ChannelOutboundHandlerAdapter

ObjectSerializerDecoder 继承MessageToMessageDecoder 间接继承ChannelInboundHandlerAdapter

RpcCmdDecoder 继承SimpleChannelInboundHandler 间接继承ChannelInboundHandlerAdapter

RpcCmdEncoder 继承MessageToMessageEncoder 间接继承ChannelOutboundHandlerAdapter

SocketManagerInitHandler 继承ChannelInboundHandlerAdapter

RpcAnswerHandler 继承SimpleChannelInboundHandler 间接继承ChannelInboundHandlerAdapter

这些handler在pipeline中的定义顺序与执行顺序是这样的，箭头是执行顺序

当一个消息从客户端写入服务端时

1、写去解决半包问题即通过定长去读取内容->LengthFieldBasedFrameDecoder

2、然后消息传递到IdleStateHandler 这样channelread被调用，正常传递到下一个handler

3、这是传递的ObjectSerializerDecoder 解码器把bytebuf转换为我们的NettyRpcCmd对象，继续到下一个handler

4、RpcCmdDecoder 会判断当前的是否是心跳消息，做不同的处理。是心跳消息则不再往下透传，不是心跳消息继续透传

5、不是心跳消息的情况下透传到SocketManagerInitHandler 这里因为IdleStateHandler的channelread被调用，则不会执行userEventTriggered方法。如果IdleStateHandler的channelread方法长时间没有调用则会定时执行userEventTriggered。

6、RpcAnswerHandler 是最后的handler,会根据消息的类型去执行不同的逻辑如果消息需要返回则会写会客户端。

当一个消息从服务端写入客户端

1、RpcCmdEncoder 对NettyRpcCmd进行编码，这是没做什么只是把NettyRpcCmd对象放入了list中，然后透传

2、到ObjectSerializerEncoder 会把NettyRpcCmd对象转化为ByteBuf,然后透传

3、然后消息传递到IdleStateHandler 这样channelread被调用，正常传递到下一个handler

4、LengthFieldPrepender 通过在消息头加入消息长度字段来解决半包问题

没想到只是一个TMRpcServer就这么多决定分开写，要不然篇幅太长