【Hystrix】【源码+图解】【六】统计功能metrics

【Hystrix】【源码+图解】【五】断路器与隔离功能

10. 统计功能metrics

前面我们分析断路器的时候，通过metrics的数据判断其是否开启

 // HystrixCircuitBreakerImpl
		@Override
        public boolean isOpen() {
            ......
            HealthCounts health = metrics.getHealthCounts();

            if (health.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
                return false;
            }

            if (health.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
                return false;
            } else {
                ......
            }
        }

    public HealthCounts getHealthCounts() {
        return healthCountsStream.getLatest();
    }

通过metrics.getHealthCounts()获取数据，那么metrics是什么？healthCountsStream又是什么？接下来我们逐一分析

HystrixCommandMetrics

先看下类图

从类图中我们可以看到主要是用XXXStream的数据结构记录数据，并且对外提供了markCommandStart()、markCommandDone()、getHealthCounts()这三个方法。metrics的工作原理：

1. 当外部调用markCommandStart()或者markCommandDone()时，发布event
2. XXXStream监听相应的event，并将其按照一定的规则统计并记录数据
3. 断路器调用getHealthCounts()方法获取数据

10.1 发布event

源码由于过于繁琐，调用markCommandStart()或者markCommandDone()的地方大概如图所示，这里我们以markCommandDone()为例，看下它是怎么产生event的

// AbstractCommand
	public Observable<R> toObservable() {
        ......
        final Action0 terminateCommandCleanup = new Action0() {
            @Override
            public void call() {
                // (2)执行结束进行清理工作
                if (_cmd.commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.TERMINAL)) {
                    handleCommandEnd(false); //user code never ran
                } else if (_cmd.commandState.compareAndSet(CommandState.USER_CODE_EXECUTED, CommandState.TERMINAL)) {
                    handleCommandEnd(true); //user code did run
                }
            }
        };
        ......
        return Observable.defer(new Func0<Observable<R>>() {
            @Override
            public Observable<R> call() {
                ......
                return afterCache
                        .doOnTerminate(terminateCommandCleanup) // (1) 声明结束时的清理工作
                        .doOnUnsubscribe(unsubscribeCommandCleanup)
                        .doOnCompleted(fireOnCompletedHook);
            }
        });
    }

    private void handleCommandEnd(boolean commandExecutionStarted) {
        
        ......
        // (3)创建处理结果
        long userThreadLatency = System.currentTimeMillis() - commandStartTimestamp;
        executionResult = executionResult.markUserThreadCompletion((int) userThreadLatency);
        ......
            // (4)创建并发布结束事件
            metrics.markCommandDone(executionResult, commandKey, threadPoolKey, commandExecutionStarted);
        ......
        }
        ......
    }

// HystrixCommandMetrics
	void markCommandDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey, boolean executionStarted) {
        HystrixThreadEventStream.getInstance().executionDone(executionResult, commandKey, threadPoolKey);
        if (executionStarted) {
            concurrentExecutionCount.decrementAndGet();
        }
    }

// HystrixThreadEventStream
	public void executionDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey) {
        // (5)创建HystrixCommandCompletion事件
        HystrixCommandCompletion event = HystrixCommandCompletion.from(executionResult, commandKey, threadPoolKey);
        writeOnlyCommandCompletionSubject.onNext(event);
    }

    private static final Action1<HystrixCommandCompletion> writeCommandCompletionsToShardedStreams = new Action1<>() {
        @Override
        public void call(HystrixCommandCompletion commandCompletion) {
            HystrixCommandCompletionStream commandStream = HystrixCommandCompletionStream.getInstance(commandCompletion.getCommandKey());
            // (6)将HystrixCommandCompletion事件发布到HystrixCommandCompletionStream流，凡是监听了它的observer就能接收到该事件
            commandStream.write(commandCompletion);
            ......
        }
    };

event发布后，监听它的observer就能收到，而这里的接收者就是xxxStream.

10.2 XXXStream

本文以HealthCountsStream为例，先看下它的类图

各项参数的意义可以参考后文它的创建

再看下其在HystrixCommandMetrics的在AbstractCommand类中的使用

// HystrixCommandMetrics
	HystrixCommandMetrics(...) {
        ......
        healthCountsStream = HealthCountsStream.getInstance(key, properties);
        ......
    }

// HealthCountsStream
	public static HealthCountsStream getInstance(HystrixCommandKey commandKey, HystrixCommandProperties properties) {
        ......
        return getInstance(commandKey, numHealthCountBuckets, healthCountBucketSizeInMs);
    }

    public static HealthCountsStream getInstance(HystrixCommandKey commandKey, int numBuckets, int bucketSizeInMs) {
        ......
            		// (1)创建
                    HealthCountsStream newStream = new HealthCountsStream(commandKey, numBuckets, bucketSizeInMs,
        ......
            // (2)启动
            healthStream.startCachingStreamValuesIfUnstarted();
            return healthStream;
        ......
    }

创建

为了方便这里把它及父类构造函数部分挪到了一起

    // numBuckets = metrics.rollingStats.timeInMilliseconds / metrics.healthSnapshot.intervalInMilliseconds
    // bucketSizeInMs = metrics.healthSnapshot.intervalInMilliseconds
	// Bucket: long[]
	// Output: HealthCountsStream.HealthCounts
    private HealthCountsStream(final HystrixCommandKey commandKey, final int numBuckets, final int bucketSizeInMs,
                               Func2<long[], HystrixCommandCompletion, long[]> reduceCommandCompletion) {
        // 设置桶的数量
        this.numBuckets = numBuckets;
       // 设置事件收集方法
        this.reduceBucketToSummary = new Func1<Observable<Event>, Observable<Bucket>>() {
            @Override
            public Observable<Bucket> call(Observable<Event> eventBucket) {
                 // 调用HystrixCommandMetrics.appendEventToBucket方法将HystrixCommandCompletion事件添加到桶里
                return eventBucket.reduce(getEmptyBucketSummary(), HystrixCommandMetrics.appendEventToBucket);
            }
        };

        // 初始化桶，设置空值
        final List<Bucket> emptyEventCountsToStart = new ArrayList<Bucket>();
        for (int i = 0; i < numBuckets; i++) {
            emptyEventCountsToStart.add(getEmptyBucketSummary());
        }

        // 将inputEventStream包装成观察对象bucketedStream
        this.bucketedStream = Observable.defer(new Func0<Observable<Bucket>>() {
            @Override
            public Observable<Bucket> call() {
                 // 即HystrixCommandCompletionStream.getInstance(commandKey)
                return inputEventStream 
                        // 获取观察对象
                        .observe() 
                        // 设置单个桶的时间窗口大小，每单位时间发射一次事件
                        .window(bucketSizeInMs, TimeUnit.MILLISECONDS) 
                        // 设置汇总数据的方法
                        .flatMap(reduceBucketToSummary)
                        // 设置初始值
                        .startWith(emptyEventCountsToStart);
            }
        });
        // 设置数据汇总方法
        Func1<Observable<Bucket>, Observable<Output>> reduceWindowToSummary = new Func1<Observable<Bucket>, Observable<Output>>() {
            @Override
            public Observable<Output> call(Observable<Bucket> window) {
                // 调用HealthCountsStream.healthCheckAccumulator方法汇总数据
                return window.scan(getEmptyOutputValue(), HealthCountsStream.healthCheckAccumulator).skip(numBuckets);
            }
        };
        // 将bucketedStream包装成观察对象sourceStream
        this.sourceStream = bucketedStream
                // 每隔1个桶的时间就发射最新numBuckets个桶的数据
                .window(numBuckets, 1) 
                // 将最新的numBuckets个桶数据汇总
                .flatMap(reduceWindowToSummary) 
                .doOnSubscribe(......)
                .doOnUnsubscribe(......)
                .share()                       
                .onBackpressureDrop();
    }

光看代码有些难理解，可以结合下面的图来看也许会相对好些

对于HystrixCommandMetrics其他的xxxStream，工作原理基本差不多如此，只不过是各数据类型不同而已，感兴趣的可以参考下表分析

构造函数里只是声明了要做什么，但是并没有开始做什么，这也就是接下来要分析的第二步-启动

启动

healthStream.startCachingStreamValuesIfUnstarted()

// BucketedCounterStream
	public void startCachingStreamValuesIfUnstarted() {
        ......
            // 1、observe()获得观察对象
            // 2、subscribe(counterSubject)，counterSubject对sourceStream进行订阅
            // 3、当sourceStream有event时，counterSubject即可接收到数据
            Subscription candidateSubscription = observe().subscribe(counterSubject);
        ......
    }

// BucketedRollingCounterStream
    @Override
    public Observable<Output> observe() {
        return sourceStream;
    }

10.3 使用数据

// HystrixCommandMetrics
	public HealthCounts getHealthCounts() {
        return healthCountsStream.getLatest();
    }

// BucketedCounterStream
	public Output getLatest() {
        startCachingStreamValuesIfUnstarted();
        if (counterSubject.hasValue()) {
            // 结果为HealthCounts
            return counterSubject.getValue();
        } else {
            return getEmptyOutputValue();
        }
    }

我们看下HealthCounts

    public static class HealthCounts {
        // 总数
        private final long totalCount;
        // 失败总数
        private final long errorCount;
        // 失败比例 (int) ((double) errorCount / totalCount * 100);
        private final int errorPercentage;
        ......
        public long getTotalRequests() {
            return totalCount;
        }

        public int getErrorPercentage() {
            return errorPercentage;
        }
        
        public HealthCounts plus(long[] eventTypeCounts) {
            long updatedTotalCount = totalCount;
            long updatedErrorCount = errorCount;

            // 统计的事件类型：SUCCESS、FAILURE、TIMEOUT、THREAD_POOL_REJECTED、SEMAPHORE_REJECTED
            long successCount = eventTypeCounts[HystrixEventType.SUCCESS.ordinal()];
            long failureCount = eventTypeCounts[HystrixEventType.FAILURE.ordinal()];
            long timeoutCount = eventTypeCounts[HystrixEventType.TIMEOUT.ordinal()];
            long threadPoolRejectedCount = eventTypeCounts[HystrixEventType.THREAD_POOL_REJECTED.ordinal()];
            long semaphoreRejectedCount = eventTypeCounts[HystrixEventType.SEMAPHORE_REJECTED.ordinal()];

            updatedTotalCount += (successCount + failureCount + timeoutCount + threadPoolRejectedCount + semaphoreRejectedCount);
            // 失败类型：除了SUCCESS以外
            updatedErrorCount += (failureCount + timeoutCount + threadPoolRejectedCount + semaphoreRejectedCount);
            return new HealthCounts(updatedTotalCount, updatedErrorCount);
        }
        ......
    }

至此，就得出了断路器判断是否开启所需的数据

 // HystrixCircuitBreakerImpl
		@Override
        public boolean isOpen() {
            ......
            HealthCounts health = metrics.getHealthCounts();

            if (health.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
                return false;
            }

            if (health.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
                return false;
            } else {
                ......
            }
        }

欢迎关注公众号三横兰