elasticsearch之索引模块简述

一，http服务监听

，NettyHttpServerTransport作为http server的角色角色。
在NettyHttpServerTransport上层Netty4HttpRequestHandler作为http请求的监听器，并将监听到的请求dispatch到NettyHttpServerTransport。

        protected void doStart() {
        boolean success = false;        
            this.serverOpenChannels = new Netty4OpenChannelsHandler(logger);
            serverBootstrap = new ServerBootstrap();
            if (blockingServer) {
                serverBootstrap.group(new OioEventLoopGroup(workerCount, daemonThreadFactory(settings,
                    HTTP_SERVER_WORKER_THREAD_NAME_PREFIX)));
                serverBootstrap.channel(OioServerSocketChannel.class);
            } else {
                serverBootstrap.group(new NioEventLoopGroup(workerCount, daemonThreadFactory(settings,
                    HTTP_SERVER_WORKER_THREAD_NAME_PREFIX)));
                serverBootstrap.channel(NioServerSocketChannel.class);
            }       
            this.boundAddress = createBoundHttpAddress();            
            success = true;        
    }

二，NettyHttpServerTransport 请求转发

Dispatches HTTP requests.

private final Dispatcher dispatcher;
    void dispatchRequest(final RestRequest request, final RestChannel channel) {
        final ThreadContext threadContext = threadPool.getThreadContext();
        try (ThreadContext.StoredContext ignore = threadContext.stashContext()) {
            dispatcher.dispatchRequest(request, channel, threadContext);
        }
    }

三，RestController负责实现Dispatcher接口，将不同的请求映射至不同的action

    @Override
    public void dispatchRequest(RestRequest request, RestChannel channel, ThreadContext threadContext) {
            ...
            final RestHandler handler = getHandler(request);
            ...
            dispatchRequest(request, responseChannel, client, threadContext, handler);
            }
        } 
    }

其中getHandler使用trie树实现了url到action的映射,其中retrieve()方法即实现了前缀查找树。在此不再贴代码。

    private RestHandler getHandler(RestRequest request) {
        String path = getPath(request);
        PathTrie<RestHandler> handlers = getHandlersForMethod(request.method());
        if (handlers != null) {
            return handlers.retrieve(path, request.params());
        } else {
            return null;
        }
    }

所有的Rest*Action都继承了抽象类BaseRestHandler，而BaseRestHandler继承了RestHandler。request即为封装的用户请求信息，handler在索引这即为RestIndexAction.

请求继续流转，BaseRestHandler实现了RestHandler中的handleRequest方法。

    public final void handleRequest(RestRequest request, RestChannel channel, NodeClient client) throws Exception {
        // prepare the request for execution; has the side effect of touching the request parameters
        final RestChannelConsumer action = prepareRequest(request, client);
        ... 
        // execute the action
        action.accept(channel);
    }

其中RestChannelConsumer为一函数式接口，具体用法为
传入一个参数，无返回值，纯消费。 方法为void accept(T t)。

prepareRequest方法的实现在RestIndexAction中，其中从RestRequest对象中获取index,type,id,routing,timestamp,version_type,op_type,ttl等参数封装为对象indexRequest。

四，NodeClient调用index()方法。

NodeClient继承了抽象类AbstractClient，AbstractClient实现了index方法，
是在此处将IndexAction进行初始化，后面会一直用到该实例。
indexAction的name属性为“indices:data/write/index”

    @Override
    public void index(final IndexRequest request, final ActionListener<IndexResponse> listener) {
        execute(IndexAction.INSTANCE, request, listener);
    }

其中doExecute()在AbstractClient方法为抽象方法，其实现类为NodeClient。

    /**
     * This is the single execution point of *all* clients.
     */
    @Override
    public final <Request extends ActionRequest, Response extends ActionResponse, RequestBuilder extends ActionRequestBuilder<Request, Response, RequestBuilder>> void execute(
            Action<Request, Response, RequestBuilder> action, Request request, ActionListener<Response> listener) {
        listener = threadedWrapper.wrap(listener);
        doExecute(action, request, listener);
    }

    protected abstract <Request extends ActionRequest, Response extends ActionResponse, RequestBuilder extends ActionRequestBuilder<Request, Response, RequestBuilder>> void doExecute(Action<Request, Response, RequestBuilder> action, Request request, ActionListener<Response> listener);

请求继续流转至NodeClient：

    /**
     * Execute an {@link Action} locally, returning that {@link Task} used to track it, and linking an {@link ActionListener}. Prefer this
     * method if you don't need access to the task when listening for the response. This is the method used to implement the {@link Client}
     * interface.
     */
    public <    Request extends ActionRequest,
                Response extends ActionResponse
            > Task executeLocally(GenericAction<Request, Response> action, Request request, ActionListener<Response> listener) {
        return transportAction(action).execute(request, listener);
    }

五，请求流转至TransportAction

TransportAction是个通用的抽象类。具体业务逻辑实现在子类或孙子类中。

TransportAction ->
         TransportReplicationAction ->
                  TransportWriteAction ->    
                          TransportSingleItemBulkWriteAction 
RestBulkAction ->
            TransportBulkAction ->
                  TransportShardBulkAction

    /**
     * Use this method when the transport action should continue to run in the context of the current task
     */
    public final void execute(Task task, Request request, ActionListener<Response> listener) {

        ...

        RequestFilterChain<Request, Response> requestFilterChain = new RequestFilterChain<>(this, logger);
        requestFilterChain.proceed(task, actionName, request, listener);
    }

其中RequestFilterChain为TransportAction的静态内部类，在这里进行一定的过滤逻辑

        public void proceed(Task task, String actionName, Request request, ActionListener<Response> listener) {
            int i = index.getAndIncrement();
            try {
                if (i < this.action.filters.length) {
                    this.action.filters[i].apply(task, actionName, request, listener, this);
                } else if (i == this.action.filters.length) {
                    this.action.doExecute(task, request, listener);
                } else {
                    listener.onFailure(new IllegalStateException("proceed was called too many times"));
                }
            } ...
        }

TransportSingleItemBulkWriteAction为抽象类TransportAction的一个实现。

    @Override
    protected void doExecute(Task task, final Request request, final ActionListener<Response> listener) {
        bulkAction.execute(task, toSingleItemBulkRequest(request), wrapBulkResponse(listener));
    }

继续调用抽象类中的execute方法
调用完毕后，请求将走向类TransportBulkAction中的doExecute()方法。核心：

    protected void doExecute(Task task, BulkRequest bulkRequest, ActionListener<BulkResponse> listener) {
        ....

        final long startTime = relativeTime();
        final AtomicArray<BulkItemResponse> responses = new AtomicArray<>(bulkRequest.requests.size());
        //如果不需要检查，即直接走创建索引，类似流式日志
        if (needToCheck()) {
            // Attempt to create all the indices that we're going to need during the bulk before we start.
            // Step 1: collect all the indices in the request
            final Set<String> indices = bulkRequest.requests.stream()
                .map(DocWriteRequest::index)
                .collect(Collectors.toSet());
            /* Step 2: filter that to indices that don't exist and we can create. At the same time build a map of indices we can't create
             * that we'll use when we try to run the requests. */
            final Map<String, IndexNotFoundException> indicesThatCannotBeCreated = new HashMap<>();
            Set<String> autoCreateIndices = new HashSet<>();
            ClusterState state = clusterService.state();
            for (String index : indices) {
                boolean shouldAutoCreate;
                try {
                    shouldAutoCreate = shouldAutoCreate(index, state);
                } catch (IndexNotFoundException e) {
                    shouldAutoCreate = false;
                    indicesThatCannotBeCreated.put(index, e);
                }
                if (shouldAutoCreate) {
                    autoCreateIndices.add(index);
                }
            }
            // Step 3: create all the indices that are missing, if there are any missing. start the bulk after all the creates come back.
            if (autoCreateIndices.isEmpty()) {
                executeBulk(task, bulkRequest, startTime, listener, responses, indicesThatCannotBeCreated);
            } else {
                final AtomicInteger counter = new AtomicInteger(autoCreateIndices.size());
                for (String index : autoCreateIndices) {
                    createIndex(index, bulkRequest.timeout(), new ActionListener<CreateIndexResponse>() {
                        @Override
                        public void onResponse(CreateIndexResponse result) {
                            if (counter.decrementAndGet() == 0) {
                                executeBulk(task, bulkRequest, startTime, listener, responses, indicesThatCannotBeCreated);
                            }
                        }
                    ...
                    });
                }
            }
        } else {
            executeBulk(task, bulkRequest, startTime, listener, responses, emptyMap());
        }
    }

六，创建索引及shard

在该方法中会判断是否需要自动创建索引，需要自动创建索引的index会放到autoCreateIndices这个set集合中。
当该set集合不为空会调用createIndex()方法，

    void createIndex(String index, TimeValue timeout, ActionListener<CreateIndexResponse> listener) {
        CreateIndexRequest createIndexRequest = new CreateIndexRequest();
        createIndexRequest.index(index);
        createIndexRequest.cause("auto(bulk api)");
        createIndexRequest.masterNodeTimeout(timeout);
        createIndexAction.execute(createIndexRequest, listener);
    }

该execute()方法实现仍在TransportAction中。

this.action.doExecute(task, request, listener);

最终走到TransportMasterNodeAction中的doExecute()实现。

    @Override
    protected void doExecute(Task task, final Request request, ActionListener<Response> listener) {
        new AsyncSingleAction(task, request, listener).start();
    }

其中AsyncSingleAction为TransportMasterNodeAction类中的一内部类。AsyncSingleAction中doStart方法有走调用建索引逻辑。该方法会判断本地节点是否是本地节点，如不是则将该请求转发至主节点。

 protected void doStart(ClusterState clusterState) {
            final Predicate<ClusterState> masterChangePredicate = MasterNodeChangePredicate.build(clusterState);
            final DiscoveryNodes nodes = clusterState.nodes();
            if (nodes.isLocalNodeElectedMaster() || localExecute(request)) {
                // check for block, if blocked, retry, else, execute locally
               ....               
                    threadPool.executor(executor).execute(new ActionRunnable(delegate) {
                        @Override
                        protected void doRun() throws Exception {
                            masterOperation(task, request, clusterState, delegate);
                        }
                    });

            } else {
                if (nodes.getMasterNode() == null) {
                    logger.debug("no known master node, scheduling a retry");
                    retry(null, masterChangePredicate);
                } else {
                    transportService.sendRequest(nodes.getMasterNode(), actionName, request, new ActionListenerResponseHandler<Response>(listener, TransportMasterNodeAction.this::newResponse) {
                        @Override
                        public void handleException(final TransportException exp) {
                            ...
                        }
                    });
                }
            }
        }

TransportAction 
        -> HandledTransportAction  
                ->  TransportMasterNodeAction 
                        ->  TransportCreateIndexAction 

masterOperation方法在TransportMasterNodeAction为抽象方法，所有必须在主节点进行的操作都实现了masterOperation方法。

    protected abstract void masterOperation(Request request, ClusterState state, ActionListener<Response> listener) throws Exception;

   protected void masterOperation(final CreateIndexRequest request, final ClusterState state, final ActionListener<CreateIndexResponse> listener) {
        ...

        final String indexName = indexNameExpressionResolver.resolveDateMathExpression(request.index());
        createIndexService.createIndex(...);
    }

创建索引方法在MetaDataCreateIndexService中的createIndex()

MetaDataCreateIndexService服务实现了createIndex()方法

//确保字符合法，不合法则会抛出异常,如不能包含’\’, ‘/’, ‘*’, ‘?’, ‘”’, ‘<’, ‘>’, ‘|’, ’ ‘, ‘,’，不能以”_”,”+”,”-“开头，字符长度不超过255，且不能等于”.”,”..”

IndicesClusterStateService物理创建索引路径

    @Override
    public synchronized void applyClusterState(final ClusterChangedEvent event) {
        if (!lifecycle.started()) {
            return;
        }

        final ClusterState state = event.state();

        // we need to clean the shards and indices we have on this node, since we
        // are going to recover them again once state persistence is disabled (no master / not recovered)
        // TODO: feels hacky, a block disables state persistence, and then we clean the allocated shards, maybe another flag in blocks?
        if (state.blocks().disableStatePersistence()) {
            for (AllocatedIndex<? extends Shard> indexService : indicesService) {
                indicesService.removeIndex(indexService.index(), NO_LONGER_ASSIGNED,
                    "cleaning index (disabled block persistence)"); // also cleans shards
            }
            return;
        }

        updateFailedShardsCache(state);

        deleteIndices(event); // also deletes shards of deleted indices

        removeUnallocatedIndices(event); // also removes shards of removed indices

        failMissingShards(state);

        removeShards(state);   // removes any local shards that doesn't match what the master expects

        updateIndices(event); // can also fail shards, but these are then guaranteed to be in failedShardsCache



        createIndices(state);

        createOrUpdateShards(state);
    }