skywalking源码分析第六篇一CoreModule核心模块启动

CoreModule能力一览

core模块提供下表Service功能集合,并由CoreModuleProvider提供实现

Service集合通过CoreModuleProvider的生命周期方法[prepare start notifyAfterCompleted]完成组件关系处理并对外提供服务

CoreModule定义的Service能力	核心关注
	整体功能,包括GRPC和Jetty,集群节点通信,存储模型服务,注册服务等等
	负责内置GRPC和jetty服务器,并允许向服务器注册handler接收不同业务消息
	负责agent注册信息逻辑处理
	负责读取数据时的存储层缓存,降低对存储的压力
	SourceReceiver底层通过分发器指定流式数据的Processor处理器

CoreModuleProvider启动原理图

成员变量名称	作用
receiver	门面类,管理所有的DispatcherManager ,对外提供统一的receive方法
receiver.DispatcherManager	manager里面包含动态生成的Dispatcher和源码定义的 Dispatcher,是一个map{scopeId,Dispatcher} ,请求携带scopeid并查找相应的Dispatcher处理数据
oalEngine	动态生成receiver.DispatcherManager里面的Dispatcher

成员变量名称	作用
storageModels	模型存储的容器,一个模型对应一个索引
annotationScan	先注册一个AnnotationListener子类,后扫描所有classes并获取class上有AnnotationListener.annotation指定的注解类,调用AnnotationListener.notify
remoteClientManager	oapServer集群的管理器,提供集群节点信息并定时刷新
grpcServer	主要与agent进行通信
jettyServer	主要与skywalking-UI进行通信

• annotationScan是一个很好的扩展点,可以自定义Listener和注解,实现对相关class的一些额外操作功能
• oalEngine通过oal文件和freemarker模板文件生成相关class

Oapserver接收请求流程图

• 结合原理图可以类比Dispatcher与mvc的Dispatcher,服务器接收请求交由handler处理,handler调用DispatcherManager进行处理
• OapEngine主要用于根据
  

源码分析一CoreModuleProvider.prepare

• 通过oalEngine引擎动态构建Dispatcher
• 构建grpcServer和jettyServer
• 通过registerServiceImplementation注册全部Service,包括服务发现,查询等等

@Override public void prepare() throws ServiceNotProvidedException, ModuleStartException {
    通过annotationScan.registerListener注册StreamAnnotationListener
    start阶段annotationScan会扫描所有的@stream注解,在扫描完成后会通过notify调用StreamAnnotationListener的create方法完成StorageModels.models填充

    StreamAnnotationListener streamAnnotationListener = new StreamAnnotationListener(getManager());
    AnnotationScan scopeScan = new AnnotationScan();
    扫描ScopeDeclaration注解
    scopeScan.registerListener(new DefaultScopeDefine.Listener());
    try {
        扫描ScopeDeclaration.Listener指定的ScopeDeclaration注解
        调用notify通知Listener哪些class配置了该注解
        scopeScan.scan();
        oal[观测分析语言]解析引擎
        oalEngine = OALEngineLoader.get();// observability analysis language
        oalEngine.setStreamListener(streamAnnotationListener);
        根据oal动态生成Dispatcher到receiver
        oalEngine.setDispatcherListener(receiver.getDispatcherManager());
        结合 oap-server/server-bootstrap/official_analysis.oal定义以及oap-server/oal-rt/code-templates目录freemarker模板文件
        生成dispatcher到receiver中
        配置环境变量SW_OAL_ENGINE_DEBUG=TRUE可查看生成源码
        源码目录/oap-server/server-core/target/oal-rt [包括度量Metrics 度量转换器 以及分发处理器]
        oalEngine.start(getClass().getClassLoader());
    } catch (Exception e) {
        throw new ModuleStartException(e.getMessage(), e);
    }

    AnnotationScan oalDisable = new AnnotationScan();
    扫描MultipleDisable注解
    oalDisable.registerListener(DisableRegister.INSTANCE);
    oalDisable.registerListener(new DisableRegister.SingleDisableScanListener());
    try {
        oalDisable.scan();
    } catch (IOException e) {
        throw new ModuleStartException(e.getMessage(), e);
    }


    创建GRPCServer 支持grpc协议 响应agent 【server-library模块library-server子模块提供GRPCServer serverlib是一个工具包 负责提供通信相关工具】
    grpcServer = new GRPCServer(moduleConfig.getGRPCHost(), moduleConfig.getGRPCPort());
    if (moduleConfig.getMaxConcurrentCallsPerConnection() > 0) {
        grpcServer.setMaxConcurrentCallsPerConnection(moduleConfig.getMaxConcurrentCallsPerConnection());
    }
    if (moduleConfig.getMaxMessageSize() > 0) {
        grpcServer.setMaxMessageSize(moduleConfig.getMaxMessageSize());
    }
    if (moduleConfig.getGRPCThreadPoolQueueSize() > 0) {
        grpcServer.setThreadPoolQueueSize(moduleConfig.getGRPCThreadPoolQueueSize());
    }
    if (moduleConfig.getGRPCThreadPoolSize() > 0) {
        grpcServer.setThreadPoolSize(moduleConfig.getGRPCThreadPoolSize());
    }
    grpcServer.initialize();
    创建jettyServer 支持http协议,响应Skywalking-UI
    jettyServer = new JettyServer(moduleConfig.getRestHost(), moduleConfig.getRestPort(), moduleConfig.getRestContextPath(), moduleConfig.getJettySelectors());
    jettyServer.initialize();
    this.registerServiceImplementation(ConfigService.class, new ConfigService(moduleConfig));
    this.registerServiceImplementation(DownsamplingConfigService.class, new DownsamplingConfigService(moduleConfig.getDownsampling()));
    注册GRPCHandlerRegister和JettyHandlerRegister,通过addHandler增加对不同请求的处理
    this.registerServiceImplementation(GRPCHandlerRegister.class, new GRPCHandlerRegisterImpl(grpcServer));
    this.registerServiceImplementation(JettyHandlerRegister.class, new JettyHandlerRegisterImpl(jettyServer));

    this.registerServiceImplementation(IComponentLibraryCatalogService.class, new ComponentLibraryCatalogService());
    注册receiver,可将server端接收到的agent数据进行分发处理
    this.registerServiceImplementation(SourceReceiver.class, receiver);

    WorkerInstancesService instancesService = new WorkerInstancesService();
    this.registerServiceImplementation(IWorkerInstanceGetter.class, instancesService);
    this.registerServiceImplementation(IWorkerInstanceSetter.class, instancesService);
    server端集群其他节点通信工具
    this.registerServiceImplementation(RemoteSenderService.class, new RemoteSenderService(getManager()));
    模型管理容器[每一个模型映射es一个索引 类似mybatis的entity与db的table映射]
    this.registerServiceImplementation(IModelSetter.class, storageModels);
    this.registerServiceImplementation(IModelGetter.class, storageModels);
    this.registerServiceImplementation(IModelOverride.class, storageModels);

    start  agent服务注册发现相关//
    this.registerServiceImplementation(ServiceInventoryCache.class, new ServiceInventoryCache(getManager(), moduleConfig));
    this.registerServiceImplementation(IServiceInventoryRegister.class, new ServiceInventoryRegister(getManager()));

    this.registerServiceImplementation(ServiceInstanceInventoryCache.class, new ServiceInstanceInventoryCache(getManager(), moduleConfig));
    this.registerServiceImplementation(IServiceInstanceInventoryRegister.class, new ServiceInstanceInventoryRegister(getManager()));

    将endpoint 和NetworkAddress等字符串信息注册到字典中 agent通过id
    this.registerServiceImplementation(EndpointInventoryCache.class, new EndpointInventoryCache(getManager(), moduleConfig));
    this.registerServiceImplementation(IEndpointInventoryRegister.class, new EndpointInventoryRegister(getManager()));
    查询网络地址  服务  服务实例
    this.registerServiceImplementation(NetworkAddressInventoryCache.class, new NetworkAddressInventoryCache(getManager(), moduleConfig));
    this.registerServiceImplementation(INetworkAddressInventoryRegister.class, new NetworkAddressInventoryRegister(getManager()));
    end //
    skywalking - ui面板查询相关
    this.registerServiceImplementation(TopologyQueryService.class, new TopologyQueryService(getManager()));
    this.registerServiceImplementation(MetricQueryService.class, new MetricQueryService(getManager()));
    this.registerServiceImplementation(TraceQueryService.class, new TraceQueryService(getManager()));
    this.registerServiceImplementation(LogQueryService.class, new LogQueryService(getManager()));
    this.registerServiceImplementation(MetadataQueryService.class, new MetadataQueryService(getManager()));
    this.registerServiceImplementation(AggregationQueryService.class, new AggregationQueryService(getManager()));
    this.registerServiceImplementation(AlarmQueryService.class, new AlarmQueryService(getManager()));
    this.registerServiceImplementation(TopNRecordsQueryService.class, new TopNRecordsQueryService(getManager()));
    this.registerServiceImplementation(CommandService.class, new CommandService(getManager()));
    处理@Stream注解
    annotationScan.registerListener(streamAnnotationListener);
    this.remoteClientManager = new RemoteClientManager(getManager(), moduleConfig.getRemoteTimeout());
    this.registerServiceImplementation(RemoteClientManager.class, remoteClientManager);
    流式处理器的一些配置
    MetricsStreamProcessor.getInstance().setEnableDatabaseSession(moduleConfig.isEnableDatabaseSession());
    TopNStreamProcessor.getInstance().setTopNWorkerReportCycle(moduleConfig.getTopNReportPeriod());
    apdexThresholdConfig = new ApdexThresholdConfig(this);
    ApdexMetrics.setDICT(apdexThresholdConfig);
}

源码分析一CoreModuleProvider.start

• grpcServer添加通信handler以及外部中间件健康检查handler
• remoteClientManager启动,提供集群的读写能力,内部借助定时任务自动刷新集群信息
• receiver.scan扫描源码中的SourceDispatcher构建DispatcherManager内部集合
• annotationScan处理@Stream注解,完成agent请求流式处理worker链的构建
• oalEngine.notifyAllListeners向DispatcherManager添加动态生成的Dispatcher
• 注册节点到集群

@Override public void start() throws ModuleStartException {
    负责OAP集群节点之间的通信
    grpcServer.addHandler(new RemoteServiceHandler(getManager()));
    负责外部比如consul等对节点自身的服务健康检查
    grpcServer.addHandler(new HealthCheckServiceHandler());
    提供集群的读写能力,内部借助定时任务自动刷新集群信息
    remoteClientManager.start();

    try {
        除去oal引擎动态编译 这里自动扫描源码中的SourceDispatcher实现并获取其处理的Source泛型对象的scopeid
        构建scopeid到SourceDispatcher的映射
        receiver.scan();
        处理@Stream注解: 构建不同维度元祖数据的Processor stream流式worker的初始化 以及Model索引实体对象的构建
        annotationScan.scan();
        自动生成类也执行annotationScan.scan的底层逻辑 【stream流式worker的初始化 模型初始化】
        oalEngine.notifyAllListeners();
    } catch (IOException | IllegalAccessException | InstantiationException e) {
        throw new ModuleStartException(e.getMessage(), e);
    }

    集群角色为Mixed和Aggregator则参与集群注册,Receiver不参与集群注册
    if (CoreModuleConfig.Role.Mixed.name().equalsIgnoreCase(moduleConfig.getRole()) || CoreModuleConfig.Role.Aggregator.name().equalsIgnoreCase(moduleConfig.getRole())) {
        RemoteInstance gRPCServerInstance = new RemoteInstance(new Address(moduleConfig.getGRPCHost(), moduleConfig.getGRPCPort(), true));
        mixed和Aggregator两种角色的oapserver节点会加入集群【默认是mixed和Aggregator】
        this.getManager().find(ClusterModule.NAME).provider().getService(ClusterRegister.class).registerRemote(gRPCServerInstance);
    }
    注册一个配置 apdexThresholdConfig
    DynamicConfigurationService dynamicConfigurationService = getManager().find(ConfigurationModule.NAME).provider().getService(DynamicConfigurationService.class);
    dynamicConfigurationService.registerConfigChangeWatcher(apdexThresholdConfig);
}

源码分析一CoreModuleProvider.notifyAfterCompleted

• 启动服务器对外提供服务
• 启动后台线程

@Override public void notifyAfterCompleted() throws ModuleStartException {
    启动grpc服务器和jetty服务器 开始正式对外提供工作
    try {
        grpcServer.start();
        jettyServer.start();
    } catch (ServerException e) {
        throw new ModuleStartException(e.getMessage(), e);
    }
    每5秒持久化数据
    PersistenceTimer.INSTANCE.start(getManager(), moduleConfig);
    skywalking采集的数据量很大,按规则对数据集进行清理
    if (moduleConfig.isEnableDataKeeperExecutor()) {
        DataTTLKeeperTimer.INSTANCE.start(getManager(), moduleConfig);
    }
    缓存更新
    CacheUpdateTimer.INSTANCE.start(getManager());
}

总结

• 通过原理图概述CoreModuleProvider的启动
• 通过流程图概述了Handler接收请求
• 通过流程图概述了Dispatcher执行请求分发
• Processor负责作为流式处理入口
• Processor内部workers负责真正的流式处理
• workers中存在RegisterRemoteWorker,意味着请求在L1聚合和L2聚合的处理可能不在同一个oapServer节点

question: 什么是L1和L2聚合

answer: 每一个worker有一个name,name为L1则worker所做的工作为L1聚合
answer:worker一般是一个责任链,头部节点是L1,尾部节点是L2
责任链共同构成了流式处理

扩展点一oalEngine构建Dispatcher

扩展点一remoteClientManager内部更新集群节点机制

扩展点一annotationScan.scan

原理图

源码

• 判断当前class是否有listener需要关心的注解
• 对符合的注解(比如@Stream)执行listener.notify

public void scan() throws IOException {
    ClassPath classpath = ClassPath.from(this.getClass().getClassLoader());
    ImmutableSet<ClassPath.ClassInfo> classes = classpath.getTopLevelClassesRecursive("org.apache.skywalking");
    for (ClassPath.ClassInfo classInfo : classes) {
        Class<?> aClass = classInfo.load();
        for (AnnotationListenerCache listener : listeners) {
            判断当前class是否有listener需要关心的注解
            if (aClass.isAnnotationPresent(listener.annotation())) {
                listener.addMatch(aClass);
            }
        }
    }
    遍历所有matchedClass执行listener.notify
    listeners.forEach(AnnotationListenerCache::complete);
}

private class AnnotationListenerCache {
    private AnnotationListener listener;
    private List<Class<?>> matchedClass;
    private void complete() {
        matchedClass.sort(Comparator.comparing(Class::getName));
        matchedClass.forEach(aClass -> listener.notify(aClass));
    }
}

Stream注解处理

• 构建worker链用于将来处理agent请求
• worker包含L1聚合,远程通信,L2聚合等
• 构建Model添加到StorageModels

public class StreamAnnotationListener implements AnnotationListener {

    @SuppressWarnings("unchecked")
    @Override public void notify(Class aClass) {
        if (aClass.isAnnotationPresent(Stream.class)) {
            Stream stream = (Stream)aClass.getAnnotation(Stream.class);
            根据不同元组的Stream标记 创建Storage元信息,并构建Model 将来ModelInstaller 会根据Model信息执行es索引初始化
            构建entryWorkers数据的用于流式处理
            if (stream.processor().equals(InventoryStreamProcessor.class)) {
                InventoryStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
            } else if (stream.processor().equals(RecordStreamProcessor.class)) {
                RecordStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
            } else if (stream.processor().equals(MetricsStreamProcessor.class)) {
                MetricsStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
            } else if (stream.processor().equals(TopNStreamProcessor.class)) {
                TopNStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
            } else {
                throw new UnexpectedException("Unknown stream processor.");
            }
        } else {
            throw new UnexpectedException("Stream annotation listener could only parse the class present stream annotation.");
        }
    }
}