高效实时监控：异步计算任务的挑战与解决方案（一）

本文主要描述，在异步计算任务背景下，怎么高效监控节点状态

背景

1. 某业务历史计算存在多业务线、大量异步计算任务，实时监控对于之后的devops，开发测试，任务跟踪，性能调优，数据可视化有重要作用；
2. 架构部目前提供的监控组件均基于 nginx 拦截，对于大量非 http 的异步计算任务不兼容；
3. 架构部目前提供的监控组件实时性较差，只提供细粒度分钟级以上的统计。

EIS

eye in sky 缩写，以对业务代码零侵入的方式，实现灵活的、轻量的实时监控

实现

• aop 自研插件实时采集数据，采用推模式，push 远程存储；
• 底层存储为时序数据库influxdb，及定制化的聚合压缩和过期策略；
• ui为grafana提供监控大盘；
• 相关技术栈均为docker部署，接近零维护成本；
• 支持耗时、qps、p99等多指标监控；

EISDingMonitor

以 AOP方式监控异步节点任务，底层采用钉钉通知形式，实时监控任务。

自定义注解

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD,ElementType.PARAMETER})
@Inherited
public @interface EISDingMonitor {
    String node();
    String remark() default "";
    boolean input() default false;
    boolean output() default false;
    String token() default "";
    String secret() default "";
    boolean log() default true;
    boolean exceptionOnly() default false;
}

属性	含义	是否必需	默认值
node	节点名称	是	无
remark	节点备注信息	否	无
input	节点输入	否	false
output	节点输出	否	false
token	钉钉群 token	否	无
secret	钉钉群 secret	否	无
log	打印日志	否	false
exceptionOnly	仅异常时推送	否	false

切片类

@Slf4j
@Aspect
@Component
public class EISDingMonitorAspect {

    @Autowired
    private DingTalkService dingTalkService;

    private static final String appName = "xxx-xxx";

    @Value("${ding.token}")
    private String robotToken;

    @Value("${ding.secret}")
    private String robotSecret;

    private static String hostName;

    static {
        try {
            hostName = InetAddress.getLocalHost().getHostName();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    @Pointcut("@annotation(com.xxx.aspect.EISDingMonitor)")
    private void pointCut(){}


    @AfterThrowing(pointcut = "pointCut()", throwing = "e")
    public void afterThrowable(JoinPoint joinPoint, Throwable e){
        try{
            Signature signature = joinPoint.getSignature();
            MethodSignature methodSignature = (MethodSignature) signature;
            Method method = methodSignature.getMethod();
            DingMonitor ding = method.getAnnotation(EISDingMonitor.class);
            String token = ding.token().trim();
            String secret = ding.secret().trim();
            token = (token == null || token.equals("")) ? robotToken : token;
            secret = (secret == null || secret.equals("")) ? robotSecret : secret;
            if(token == null || token.equals("") || secret == null || secret.equals("")){
                return;
            }

            String node = ding.node();
            String remark = ding.remark();
            remark = remark.equals("") ? null : remark;
            long currentTime = System.currentTimeMillis();
            String inputStr;

            Object[] args = joinPoint.getArgs();

            inputStr = ding.input()
                    && Objects.nonNull(args)
                    && args.length > 0
                    ? Arrays.stream(args).map(o -> o.toString()).collect(Collectors.toList()).toString()
                    : null;

            DingTalkCardMsg failedMsg = DingTalkCardMsg.builder()
                    .reporter(appName)
                    .host(hostName)
                    .node(node)
                    .remark(remark)
                    .status("【failed】")
                    .current(DateUtil.timeStamp2Date(currentTime, "yyyy-MM-dd HH:mm:ss"))
                    .input(inputStr)
                    .msg("节点失败, e = " + e.getMessage())
                    .build();
            dingTalkService.ding(failedMsg, token, secret);

            if(ding.log()){
                log.error("EIS ding logging, threadId = {}, method = {}, input = {}, remark = {}",
                        Thread.currentThread().getId(),
                        method.getName(),
                        inputStr,
                        remark,
                        e);
            }
        } catch (Exception localException){
            log.error("failed to push eis ding msg, e = {}", localException.getMessage());
        }
    }

    @Around("pointCut()")
    public Object around(ProceedingJoinPoint joinPoint) throws Throwable {

        MethodSignature methodSignature = (MethodSignature) joinPoint.getSignature();

        Method method = methodSignature.getMethod();

        DingMonitor ding = method.getAnnotation(EISDingMonitor.class);

        String token = ding.token().trim();

        String secret = ding.secret().trim();

        token = (token == null || token.equals("")) ? robotToken : token;

        secret = (secret == null || secret.equals("")) ? robotSecret : secret;

        if (token == null || token.equals("") || secret == null || secret.equals("")) {
            return joinPoint.proceed();
        }

        if (ding.exceptionOnly()) {
            return joinPoint.proceed();
        }

        String node = ding.node();
        String remark = ding.remark();

        Object[] args = joinPoint.getArgs();

        String inputStr = null;

        try {
            inputStr = ding.input()
                    && Objects.nonNull(args)
                    && args.length > 0
                    ? Arrays.stream(args).map(o -> o.toString()).collect(Collectors.toList()).toString()
                    : null;

            remark = remark.equals("") ? null : remark;

            DingTalkCardMsg startMsg = DingTalkCardMsg.builder()
                    .reporter(appName)
                    .host(hostName)
                    .current(DateUtil.timeStamp2Date(System.currentTimeMillis(), "yyyy-MM-dd HH:mm:ss"))
                    .node(node)
                    .remark(remark)
                    .status("start")
                    .msg("进入节点")
                    .input(inputStr)
                    .build();
            dingTalkService.ding(startMsg, token, secret);
        } catch (Exception e){
            log.error("failed to push ding msg, e = {}", e.getMessage());
        }

        long startTime = System.currentTimeMillis();
        Object result = joinPoint.proceed();
        long currentTime = System.currentTimeMillis();

        try{
            String cost = DateUtil.duration(startTime, currentTime);

            String outputStr = ding.output() && !Objects.isNull(result) ? result.toString() : null;

            DingTalkCardMsg successMsg = DingTalkCardMsg.builder()
                    .reporter(appName)
                    .host(hostName)
                    .current(DateUtil.timeStamp2Date(currentTime, "yyyy-MM-dd HH:mm:ss"))
                    .node(node)
                    .remark(remark)
                    .status("success")
                    .msg("节点完成")
                    .cost(cost)
                    .input(inputStr)
                    .output(outputStr)
                    .build();
            dingTalkService.ding(successMsg, token, secret);

            if(ding.log()){
                log.info("monitor ding logging, threadId = {}, method = {}, cost = {}, input = {}, output = {}, remark = {}",
                        Thread.currentThread().getId(),
                        method.getName(),
                        cost,
                        inputStr,
                        outputStr,
                        remark);
            }
        }catch (Exception e){
            log.error("failed to push monitor ding msg, e = {}", e.getMessage());
        }
        return result;
    }
}

总结

通过 AOP 切面，可以实时观察到任务的状态，包括执行到哪一步、输入参数、输出参数、执行成功还是失败、耗时等维度信息。

关于influxdb存储的部分，则是另一个注解EISTimerMonitor了，关于这个的实现，我们下次再说。