Solr4.7.2启动时的Index locked for write for core问题分析

Solr在启动时，通过多线程的方式加载core，在加载完每个core的配置文件后，实例化了一个SolrCore，在SolrCore的构造函数中，会初始化index，Index locked for write for core的Exception就是在这个时候报出来的，Solr启动过程和SolrCore的构造函数可看之前的文章：

http://blog.csdn.net/wenchanter/article/details/37603621

此处主要看下引发Index locked for write for core错误的代码：

首先看下SolrCore构造函数中的initIndex方法：

  void initIndex(boolean reload) throws IOException {
      // 着里面用dataDir生成了一个Directory，并设置了锁工厂
      String indexDir = getNewIndexDir();
      boolean indexExists = getDirectoryFactory().exists(indexDir);
      boolean firstTime;
      synchronized (SolrCore.class) {
        firstTime = dirs.add(getDirectoryFactory().normalize(indexDir));
      }
      boolean removeLocks = solrConfig.unlockOnStartup;

      // solrconfig.xml中得indexReaderFactory,若不配置，也会默认StandardIndexReaderFactory
      initIndexReaderFactory();

      if (indexExists && firstTime && !reload) {
        // 除了上面那个getNewIndexDir中的用dataDir创建的Directory设置了lockFactory，这里又用其返回的index路径创建了一个Directory，并设置了一遍lockFactory，真正判断是否加锁的时用这个Directory
        Directory dir = directoryFactory.get(indexDir, DirContext.DEFAULT,
            getSolrConfig().indexConfig.lockType);
        try {
          if (IndexWriter.isLocked(dir)) {
            if (removeLocks) {
              log.warn(
                  logid
                      + "WARNING: Solr index directory '{}' is locked.  Unlocking...",
                  indexDir);
              IndexWriter.unlock(dir);
            } else {
              log.error(logid
                  + "Solr index directory '{}' is locked.  Throwing exception",
                  indexDir);
              throw new LockObtainFailedException(
                  "Index locked for write for core " + name);
            }
           
          }
        } finally {
          directoryFactory.release(dir);
        }
      }

      // Create the index if it doesn't exist.
      if(!indexExists) {
        log.warn(logid+"Solr index directory '" + new File(indexDir) + "' doesn't exist."
                + " Creating new index...");

        SolrIndexWriter writer = SolrIndexWriter.create("SolrCore.initIndex", indexDir, getDirectoryFactory(), true,
                                                        getLatestSchema(), solrConfig.indexConfig, solrDelPolicy, codec);
        writer.close();
      }
  }

可见是用IndexWriter.isLocked(dir)判断是否加锁，如果已经加了锁，则分为两种情况，一种是在solrconfig.xml中配置了unlockOnStartup，会尝试unlock，如果没有配置unlockStartup，则会抛出Index locked for write for core异常。

这个被校验的dir是一个Directory，这个Directory是通过

Directory   dir =   directoryFactory .get(indexDir,   DirContext . DEFAULT ,
            getSolrConfig(). indexConfig . lockType );

得到的，其中indexDir是上面取到的$solrhome/collection1/data/index索引存放的文件夹，directoryFactory此处是NRTCachingDirectoryFactory实例，看下其中的get方法：

  @Override
  public final Directory get(String path,  DirContext dirContext, String rawLockType)
      throws IOException {
    String fullPath = normalize(path);
    synchronized (this) {
      if (closed) {
        throw new AlreadyClosedException("Already closed");
      }
     
      final CacheValue cacheValue = byPathCache.get(fullPath);
      Directory directory = null;
      if (cacheValue != null) {
        directory = cacheValue.directory;
      }
     
      if (directory == null) { 
        // 由于是NRTCachingDirectoryFactory，这里面直接new了一个NTRCachingDirectory，里面的FSDirectory.open(File)的方法根据不同的系统new不同的FSDirectory.
        directory = create(fullPath, dirContext);
       
        directory = rateLimit(directory);
       
        CacheValue newCacheValue = new CacheValue(fullPath, directory);
        // 这里设置锁工厂
        injectLockFactory(directory, fullPath, rawLockType);
       
        byDirectoryCache.put(directory, newCacheValue);
        byPathCache.put(fullPath, newCacheValue);
        log.info("return new directory for " + fullPath);
      } else {
        cacheValue.refCnt++;
        log.debug("Reusing cached directory: {}", cacheValue);
      }
     
      return directory;
    }
  }

get方法中的injectLockFactory设置了锁工厂，根据不同的属性设置不同的锁工厂，此处配置的是native，因此设置了一个NativeFSLockFactory：

  private static Directory injectLockFactory(Directory dir, String lockPath,
      String rawLockType) throws IOException {
    if (null == rawLockType) {
      // we default to "simple" for backwards compatibility
      log.warn("No lockType configured for " + dir + " assuming 'simple'");
      rawLockType = "simple";
    }
    final String lockType = rawLockType.toLowerCase(Locale.ROOT).trim();
   
    if ("simple".equals(lockType)) {
      // multiple SimpleFSLockFactory instances should be OK
      dir.setLockFactory(new SimpleFSLockFactory(lockPath));
    } else if ("native".equals(lockType)) {
      dir.setLockFactory(new NativeFSLockFactory(lockPath));
    } else if ("single".equals(lockType)) {
      if (!(dir.getLockFactory() instanceof SingleInstanceLockFactory)) dir
          .setLockFactory(new SingleInstanceLockFactory());
    } else if ("hdfs".equals(lockType)) {
      Directory del = dir;
     
      if (dir instanceof NRTCachingDirectory) {
        del = ((NRTCachingDirectory) del).getDelegate();
      }
     
      if (del instanceof BlockDirectory) {
        del = ((BlockDirectory) del).getDirectory();
      }
     
      if (!(del instanceof HdfsDirectory)) {
        throw new SolrException(ErrorCode.FORBIDDEN, "Directory: "
            + del.getClass().getName()
            + ", but hdfs lock factory can only be used with HdfsDirectory");
      }

      dir.setLockFactory(new HdfsLockFactory(((HdfsDirectory)del).getHdfsDirPath(), ((HdfsDirectory)del).getConfiguration()));
    } else if ("none".equals(lockType)) {
      // Recipe for disaster
      log.error("CONFIGURATION WARNING: locks are disabled on " + dir);
      dir.setLockFactory(NoLockFactory.getNoLockFactory());
    } else {
      throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
          "Unrecognized lockType: " + rawLockType);
    }
    return dir;
  }

上面再new NativeFSLockFactory时，构造函数中调用父类FSLockFactory中的setLockDir(File lockDir)方法，这个方法只能调用一次，另外，在dir.setLockFactory时，也做了一些工作，看下一连串的方法调用：

NTRCachingDirectory中的setLockFactory:
  @Override
  public void setLockFactory(LockFactory lf) throws IOException {
    // 这里是了NIOFSDirectory
    delegate.setLockFactory(lf);
  }
这里调用的是FSDirectory的setLockFactory:
  @Override
  public void setLockFactory(LockFactory lockFactory) throws IOException {
    super.setLockFactory(lockFactory);

    // for filesystem based LockFactory, delete the lockPrefix, if the locks are placed
    // in index dir. If no index dir is given, set ourselves
    if (lockFactory instanceof FSLockFactory) {
      final FSLockFactory lf = (FSLockFactory) lockFactory;
      final File dir = lf.getLockDir();
      // if the lock factory has no lockDir set, use the this directory as lockDir
      if (dir == null) {
        lf.setLockDir(directory);
        lf.setLockPrefix(null);
      } else if (dir.getCanonicalPath().equals(directory.getCanonicalPath())) {
        lf.setLockPrefix(null);
      }
    }

  }
super.setLockFactory是指父类BaseDirectory：
  @Override
  public void setLockFactory(LockFactory lockFactory) throws IOException {
    assert lockFactory != null;
    this.lockFactory = lockFactory;
    lockFactory.setLockPrefix(this.getLockID());
  }
然后着里面的getLockID方法又使调用的FSDirectory中的,看看是不是可以发现LockID计算方法和String的hash算法是一样的：
  @Override
  public String getLockID() {
    ensureOpen();
    String dirName;                               // name to be hashed
    try {
      dirName = directory.getCanonicalPath();
    } catch (IOException e) {
      throw new RuntimeException(e.toString(), e);
    }

    int digest = 0;
    for(int charIDX=0;charIDX<dirName.length();charIDX++) {
      final char ch = dirName.charAt(charIDX);
      digest = 31 * digest + ch;
    }
    return "lucene-" + Integer.toHexString(digest);
  }

看完了设置锁工厂，再来看下校验锁的方法：

IndexWriter.isLocked(dir);

  /**
   * Returns <code>true</code> iff the index in the named directory is
   * currently locked.
   * @param directory the directory to check for a lock
   * @throws IOException if there is a low-level IO error
   */
  public static boolean isLocked(Directory directory) throws IOException {
    return directory.makeLock(WRITE_LOCK_NAME).isLocked();
  }

其中makeLock只是返回了一个NativeFSLock实例，WRITE_LOCK_NAME是默认的“write.lock”，看下NativeFSLock的isLocked()方法：

  @Override
  public synchronized boolean isLocked() {
    // The test for is isLocked is not directly possible with native file locks:
   
    // First a shortcut, if a lock reference in this instance is available
    if (lockExists()) return true;
   
    // Look if lock file is present; if not, there can definitely be no lock!
    if (!path.exists()) return false;
   
    // Try to obtain and release (if was locked) the lock
    try {
      boolean obtained = obtain();
      if (obtained) close();
      return !obtained;
    } catch (IOException ioe) {
      return false;
    }   
  }

看下obtain()方法，这时的lockDir是目录$solrhome/collection1/data/index，这个属性是当时在创建Directory时setLockFactory时，新建NativeFSLock实例的时候设置的，就是那个只能设置一次的方法，而obtain后面的close方法，会释放锁：

  @Override
  public synchronized boolean obtain() throws IOException {

    if (lockExists()) {
      // Our instance is already locked:
      return false;
    }

    // Ensure that lockDir exists and is a directory.
    if (!lockDir.exists()) {
      if (!lockDir.mkdirs())
        throw new IOException("Cannot create directory: " +
                              lockDir.getAbsolutePath());
    } else if (!lockDir.isDirectory()) {
      // TODO: NoSuchDirectoryException instead?
      throw new IOException("Found regular file where directory expected: " +
                            lockDir.getAbsolutePath());
    }
    // path是个File，/Users/wanghui/Documents/workspace/lucene_solr_4_7_2/solr/solrweb/solrhome/collection1/data/index/write.lock
    String canonicalPath = path.getCanonicalPath();

    boolean markedHeld = false;

    try {

      // Make sure nobody else in-process has this lock held
      // already, and, mark it held if not:

      synchronized(LOCK_HELD) {
        if (LOCK_HELD.contains(canonicalPath)) {
          // Someone else in this JVM already has the lock:
          return false;
        } else {
          // This "reserves" the fact that we are the one
          // thread trying to obtain this lock, so we own
          // the only instance of a channel against this
          // file:
          LOCK_HELD.add(canonicalPath);
          markedHeld = true;
        }
      }

      try {
        f = new RandomAccessFile(path, "rw");
      } catch (IOException e) {
        // On Windows, we can get intermittent "Access
        // Denied" here.  So, we treat this as failure to
        // acquire the lock, but, store the reason in case
        // there is in fact a real error case.
        failureReason = e;
        f = null;
      }

      if (f != null) {
        try {
          channel = f.getChannel();
          try {
            lock = channel.tryLock();
          } catch (IOException e) {
            // At least on OS X, we will sometimes get an
            // intermittent "Permission Denied" IOException,
            // which seems to simply mean "you failed to get
            // the lock".  But other IOExceptions could be
            // "permanent" (eg, locking is not supported via
            // the filesystem).  So, we record the failure
            // reason here; the timeout obtain (usually the
            // one calling us) will use this as "root cause"
            // if it fails to get the lock.
            failureReason = e;
          } finally {
            if (lock == null) {
              try {
                channel.close();
              } finally {
                channel = null;
              }
            }
          }
        } finally {
          if (channel == null) {
            try {
              f.close();
            } finally {
              f = null;
            }
          }
        }
      }

    } finally {
      if (markedHeld && !lockExists()) {
        synchronized(LOCK_HELD) {
          if (LOCK_HELD.contains(canonicalPath)) {
            LOCK_HELD.remove(canonicalPath);
          }
        }
      }
    }
    return lockExists();
  }

可以看到其中LOCK_HELD这个set如果里面包含这个路径也会直接返回false，如果LOCK_HELD不包含此路径，则会尝试获得锁。

从上面的分析可以看出，此处只是校验是否可以加锁，而真正加锁的地方，其实是在后面实例化SolrIndexWriter的时候，SolrIndeWriter调用父类IndexWriter的构造函数的时候加的锁，具体过程可见solr启动分析的这篇文章：
http://blog.csdn.net/wenchanter/article/details/37743977