solr-6.1.0源码分析---1

solr-6.1.0源码分析—SolrDispatchFilter初始化

SolrDispatchFilter继承自javax.servlet.Filter，并且在web.xml里有如下一段配置。

  <filter>
    <filter-name>SolrRequestFilter</filter-name>
    <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
    <init-param>
      <param-name>excludePatterns</param-name>
      <param-value>/css/.+,/js/.+,/img/.+,/tpl/.+</param-value>
    </init-param>
  </filter>

因此，经过tomcat框架的处理后，最先会调用SolrRequestFilter的init函数。

SolrDispatchFilter::init

  public void init(FilterConfig config) throws ServletException{

    String exclude = config.getInitParameter("excludePatterns");
    if(exclude != null) {
      String[] excludeArray = exclude.split(",");
      excludePatterns = new ArrayList<>();
      for (String element : excludeArray) {
        excludePatterns.add(Pattern.compile(element));
      }
    }

    this.cores = createCoreContainer(SolrResourceLoader.locateSolrHome(), new Properties());
    this.httpClient = cores.getUpdateShardHandler().getHttpClient();
  }

首先将web.xml中配置的/css/.+、/js/.+、/img/.+和/tpl/.+添加到excludePatterns列表中。然后通过createCoreContainer创建CoreContainer并获取HttpClient。

SolrDispatchFilter::init->createCoreContainer

  protected CoreContainer createCoreContainer(Path solrHome, Properties extraProperties) {
    NodeConfig nodeConfig = loadNodeConfig(solrHome, extraProperties);
    cores = new CoreContainer(nodeConfig, extraProperties, true);
    cores.load();
    return cores;
  }

loadNodeConfig根据solrHome目录下的solr.xml文件生成NodeConfig，然后创建CoreContainer封装NodeConfig信息，最后通过load函数继续进行初始化，下面一一来看。

loadNodeConfig

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig

  public static NodeConfig loadNodeConfig(Path solrHome, Properties nodeProperties) {

    SolrResourceLoader loader = new SolrResourceLoader(solrHome, null, nodeProperties);

    ...
    return SolrXmlConfig.fromSolrHome(loader, loader.getInstancePath());
  }

省略掉的部分和zookeeper相关，本章不考虑。首先创建SolrResourceLoader，然后通过fromSolrHome函数解析solrhome目录下的solr.xml配置文件并生成NodeConfig。

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig->SolrResourceLoader::SolrResourceLoader

  public SolrResourceLoader(Path instanceDir, ClassLoader parent, Properties coreProperties) {

    if (parent == null)
      parent = Thread.currentThread().getContextClassLoader();
    this.classLoader = new URLClassLoader(new URL[0], parent);

    ...
    this.coreProperties = coreProperties;
  }

SolrResourceLoader构造函数主要创建了URLClassLoader，用来加载Class文件。

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig->SolrXmlConfig::fromSolrHome

  public static NodeConfig fromSolrHome(SolrResourceLoader loader, Path solrHome) {
    return fromFile(loader, solrHome.resolve(SOLR_XML_FILE));
  }

  public static NodeConfig fromFile(SolrResourceLoader loader, Path configFile) {
    InputStream inputStream = Files.newInputStream(configFile);
    return fromInputStream(loader, inputStream);
  }

SOLR_XML_FILE默认为solrhome目录下的solr.xml，fromSolrHome函数将solr.xml文件变为输入流调用fromInputStream继续解析。首先看一下solr.xml的默认配置。

<solr>
  <solrcloud>
    <str name="host">${host:}</str>
    <int name="hostPort">${jetty.port:8983}</int>
    <str name="hostContext">${hostContext:solr}</str>
    <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
    <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
    <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
    <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
    <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
    <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
  </solrcloud>

  <shardHandlerFactory name="shardHandlerFactory"
    class="HttpShardHandlerFactory">
    <int name="socketTimeout">${socketTimeout:600000}</int>
    <int name="connTimeout">${connTimeout:60000}</int>
  </shardHandlerFactory>
</solr>

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig->SolrXmlConfig::fromSolrHome->fromFile->fromInputStream

  public static NodeConfig fromInputStream(SolrResourceLoader loader, InputStream is) {
    byte[] buf = IOUtils.toByteArray(is);
    ByteArrayInputStream dup = new ByteArrayInputStream(buf);
    Config config = new Config(loader, null, new InputSource(dup), null, false);
    return fromConfig(config);
  }

根据文件内容创建Config，Config构造函数将solr.xml中的配置信息解析成Document文档，然后通过fromConfig

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig->SolrXmlConfig::fromSolrHome->fromFile->fromInputStream->Config::Config

  public Config(SolrResourceLoader loader, String name, InputSource is, String prefix, boolean substituteProps) throws ParserConfigurationException, IOException, SAXException
  {

      javax.xml.parsers.DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();

      final DocumentBuilder db = dbf.newDocumentBuilder();
      db.setEntityResolver(new SystemIdResolver(loader));
      db.setErrorHandler(xmllog);

      doc = db.parse(is);
      origDoc = copyDoc(doc);
  }

DocumentBuilder的parse函数将文档输入流转变为Document对象。

SolrDispatchFilter::init->createCoreContainer->loadNodeConfig->SolrXmlConfig::fromSolrHome->fromFile->fromInputStream->fromConfig

  public static NodeConfig fromConfig(Config config) {

    config.substituteProperties();

    CloudConfig cloudConfig = null;
    UpdateShardHandlerConfig deprecatedUpdateConfig = null;

    if (config.getNodeList("solr/solrcloud", false).getLength() > 0) {
      NamedList<Object> cloudSection = readNodeListAsNamedList(config, "solr/solrcloud/*[@name]", "<solrcloud>");
      deprecatedUpdateConfig = loadUpdateConfig(cloudSection, false);
      cloudConfig = fillSolrCloudSection(cloudSection);
    }
    updateConfig = deprecatedUpdateConfig;

    NodeConfig.NodeConfigBuilder configBuilder = new NodeConfig.NodeConfigBuilder(nodeName, config.getResourceLoader());
    configBuilder.setUpdateShardHandlerConfig(updateConfig);
    configBuilder.setShardHandlerFactoryConfig(getShardHandlerFactoryPluginInfo(config));
    configBuilder.setLogWatcherConfig(loadLogWatcherConfig(config, "solr/logging/*[@name]", "solr/logging/watcher/*[@name]"));
    configBuilder.setSolrProperties(loadProperties(config));
    if (cloudConfig != null)
      configBuilder.setCloudConfig(cloudConfig);
    return fillSolrSection(configBuilder, entries);
  }

substituteProperties将一些属性由solr.propertyies属性替换。
如果solr.xml配置文件中存在solrcloud标签，则首先通过readNodeListAsNamedList函数将其中的键值对读取到cloudSection中，然后继续通过loadUpdateConfig函数从cloudSection中读取有关更新的数据并创建UpdateShardHandlerConfig用来封装这些信息，最后通过fillSolrCloudSection函数从cloudSection中提取出有关Cloud的配置并存储在CloudConfig中。
fromConfig函数接下来再次调用readNodeListAsNamedList函数读取到shardHandlerFactory标签的信息至entries中。
接下来创建NodeConfigBuilder并添加刚刚读取出来的各个信息，最后在fillSolrSection函数中会调用其build函数返回一个NodeConfig。

Load

创建完NodeConfig后，会创建CoreContainer并调用其load函数，根据NodeConfig中的配置信息创建相应的Handler。

SolrDispatchFilter::init->createCoreContainer->CoreContainer::load

  public void load()  {
    String libDir = cfg.getSharedLibDirectory();
    Path libPath = loader.getInstancePath().resolve(libDir);
    loader.addToClassLoader(SolrResourceLoader.getURLs(libPath));
    loader.reloadLuceneSPI();

    shardHandlerFactory = ShardHandlerFactory.newInstance(cfg.getShardHandlerFactoryPluginInfo(), loader);
    updateShardHandler = new UpdateShardHandler(cfg.getUpdateShardHandlerConfig());
    solrCores.allocateLazyCores(cfg.getTransientCacheSize(), loader);
    logging = LogWatcher.newRegisteredLogWatcher(cfg.getLogWatcherConfig(), loader);
    hostName = cfg.getNodeName();

    ...

    containerHandlers.put(ZK_PATH, new ZookeeperInfoHandler(this));
    securityConfHandler = new SecurityConfHandler(this);
    collectionsHandler = createHandler(cfg.getCollectionsHandlerClass(), CollectionsHandler.class);
    containerHandlers.put(COLLECTIONS_HANDLER_PATH, collectionsHandler);
    infoHandler        = createHandler(cfg.getInfoHandlerClass(), InfoHandler.class);
    containerHandlers.put(INFO_HANDLER_PATH, infoHandler);
    coreAdminHandler   = createHandler(cfg.getCoreAdminHandlerClass(), CoreAdminHandler.class);
    containerHandlers.put(CORES_HANDLER_PATH, coreAdminHandler);
    configSetsHandler = createHandler(cfg.getConfigSetsHandlerClass(), ConfigSetsHandler.class);
    containerHandlers.put(CONFIGSETS_HANDLER_PATH, configSetsHandler);
    containerHandlers.put(AUTHZ_PATH, securityConfHandler);
    containerHandlers.put(AUTHC_PATH, securityConfHandler);
    if(pkiAuthenticationPlugin != null)
      containerHandlers.put(PKIAuthenticationPlugin.PATH, pkiAuthenticationPlugin.getRequestHandler());

    coreConfigService = ConfigSetService.createConfigSetService(cfg, loader, zkSys.zkController);
    containerProperties.putAll(cfg.getSolrProperties());

    ...

    List<CoreDescriptor> cds = coresLocator.discover(this);

    ...

  }

libDir是solrhome目录下lib的相对路径，libPath是lib的绝对路径。首先通过addToClassLoader函数将该lib目录添加到URLClassLoader中，然后通过reloadLuceneSPI重新加载lucene的核心类文件。
load函数接下来初始化各个成员变量，其中最重要的是创建各个用来处理http请求的Handler并添加到containerHandlers列表中。
其中在创建UpdateShardHandler时，创建了HttpClient。

  public UpdateShardHandler(UpdateShardHandlerConfig cfg) {
    this.cfg = cfg;
    clientConnectionManager = new PoolingClientConnectionManager(SchemeRegistryFactory.createSystemDefault());
    clientConnectionManager.setMaxTotal(cfg.getMaxUpdateConnections());
    clientConnectionManager.setDefaultMaxPerRoute(cfg.getMaxUpdateConnectionsPerHost());

    ModifiableSolrParams clientParams = getClientParams();
    client = HttpClientUtil.createClient(clientParams, clientConnectionManager);
  }

  public static CloseableHttpClient createClient(final SolrParams params, ClientConnectionManager cm) {
    final ModifiableSolrParams config = new ModifiableSolrParams(params);
    final DefaultHttpClient httpClient = HttpClientFactory.createHttpClient(cm);
    configureClient(httpClient, config);
    return httpClient;
  }

HttpClientFactory的createHttpClient函数创建了DefaultHttpClient，然后通过configureClient进行配置。

SolrDispatchFilter::init->createCoreContainer->CoreContainer::load->CorePropertiesLocator::discover

  public List<CoreDescriptor> discover(final CoreContainer cc) {
    final List<CoreDescriptor> cds = Lists.newArrayList();
    Set<FileVisitOption> options = new HashSet<>();
    options.add(FileVisitOption.FOLLOW_LINKS);
    final int maxDepth = 256;

    Files.walkFileTree(this.rootDirectory, options, maxDepth, new SimpleFileVisitor<Path>() {
      @Override
      public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
        if (file.getFileName().toString().equals(PROPERTIES_FILENAME)) {
          CoreDescriptor cd = buildCoreDescriptor(file, cc);
          cds.add(cd);
          return FileVisitResult.SKIP_SIBLINGS;
        }
        return FileVisitResult.CONTINUE;
      }
    });

    return cds;
  }

PROPERTIES_FILENAME为core.properties，在solrhome目录下查找core.properties文件。
buildCoreDescriptor函数将其中的key和value封装成CoreDescriptor添加到cds列表中并返回。
下面假设该cds列表为空。