一.环境启动代码
SolrResourceLoader.SetSolrHome("/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr");
SolrResourceLoader.SetSchemaHome("/home/taobao/solr/schema");
ShardPartion.base = "/group/tbdp-etao-adhoc/p4padhoc/tabletest"; //必须的,如果不配置会有异常,SolrCore中会使用ShardPartion.getHdfsRealtimePath
ShardPartion.taskIndex = 1;
ShardPartion.index = IndexUtils.getHdfsForder(1);
String hdfsconf = "/duitang/dist/sys/hadoop-1.2.1/conf/"; //必须的,如果不配置会有异常,ReadOnlyDirectory.getConf()
HadoopUtil.setHdfsConfDir(hdfsconf);
JettySolrRunner jetty = new JettySolrRunner("/solr", 1210);
jetty.start();
while (true)
{
Thread.sleep(1000);
}
二.启动之后通过URL测试
http://localhost:1210/solr/select?q=*:*&indexpartion=20111014
三.查询入口SolrDispatchFilter
入口在SolrDispatchFilter中,通过拦截请求中的select来加载对应的requestHandler.代码如下:
init():
CoreContainer.Initializer init = new CoreContainer.Initializer();
CoreContainer cores = init.initialize();
core = cores.getCore("");
doFilter():
final SolrConfig config = core.getSolrConfig();
SolrRequestParsers parser = new SolrRequestParsers(config);
SolrQueryRequest solrReq = parser.parse( core, "select", httpServletRequest)
SolrRequestHandler handler = core.getRequestHandler(null)
SolrQueryResponse solrRsp = new SolrQueryResponse();
solrReq.getContext().put( "webapp", req.getContextPath() );
solrReq.getCore().execute( handler, solrReq, solrRsp );
四.核心方法SolrCore.execute()
入口SolrCore.execute(),前面都是web请求忽略掉。
SolrCore.execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryResponse rsp)执行过程:
SolrCore.execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryResponse rsp)
-->handler.handleRequest(req, rsp);
-->setResponseHeaderValues(handler, req, rsp);
SolrCore实际调用的是SearchHandler.handleRequest(),SearchHandler继承于RequestHandlerBase,覆盖了handleRequestBody(),调用关系如下:
-->RequestHandlerBase.handleRequest(SolrQueryRequest req, SolrQueryResponse rsp)
-->SearchHandler.handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)
SearchHandler.handleRequestBody的实现比较复杂,如果shards不为null,会尝试请求各shards然后对结果合并。这里先看shards为null的情况:
-->QueryComponent.process(ResponseBuilder rb)
-->SolrIndexSearcher searcher = rb.req.getSearcher();
-->searcher.search(result,cmd);
-->FacetComponent.process(ResponseBuilder rb)
req是SolrQueryRequest类型,实际是SolrQueryRequestBase实例,看看其getSearcher()实现方法。
1.SolrQueryRequestBase.getSearcher()
-->String partion=this.getParams().get(CommonParams.PARTION,"default");
-->SolrCore.getSearcher(partion,false,false,higohb);
2.SolrCore.getSearcher(String partion, boolean mdirs, boolean isclearCache, boolean ishb
-->String partionKey=this.corename+"@"+partion+"@"+SolrResourceLoader.getCacheFlushKey(p);
-->SolrCore.getSearcherByPath(p,partionKey, f.getAbsolutePath(), mdirs, isclearCache, ishb)
3.SolrCore.getSearcherByPath(PartionKey p, String partionKey, String path):
if partion.name == "default":
//初始化IndexReader
RAMDirectory rd=new RAMDirectory();
rd.setCore(this,p);
IndexWriter writer=new IndexWriter(rd, null,new KeepOnlyLastCommitDeletionPolicy(), MaxFieldLength.UNLIMITED);
writer.setMergeFactor(10);
writer.setUseCompoundFile(false);
writer.close();
reader=IndexReader.open(rd);
//初始化SolrIndexSearcher
SolrIndexSearcher newSearcher = new SolrIndexSearcher(this, schema, "partion_" + partionKey, reader, true);
newSearcher.setPartionCacheKey(p);
newSearcher.setFieldcacheDir(fieldcacheDir);
newSearcher.setPartionKey(partionKey);
rtn = newHolderPartion(newSearcher);
searchCache.put(partionKey, rtn);
else:
//初始化IndexReader
MdrillDirectory rtdir = new ReadOnlyDirectory(f, HadoopUtil.hadoopConfDir, ShardPartion.getHdfsRealtimePath(p.tablename, p.partion));
rtdir.getForSearch()
fs.mkdirs(new Path(hdfsPath).getParent()); //hdfs_path--> /group/.../20110111
path.mkdirs(); //local_path--> /duitang/.../20110111
File links=new File(path,"indexLinks");
while f=links.readline():
FSDirectory d=LinkFSDirectory.open(f);
DirectoryInfo info=new DirectoryInfo();
info.d=d;
info.tp=DirectoryInfo.DirTpe.file;
diskDirector.put(f, info);
IndexReader[] r = new IndexReader[diskDirector.size()];
for directory in diskDirector.values:
r[i] = IndexReader.open(directory)
if r.length==1:
reader = r[0]
else:
reader = new MultiReader(r, true);
//初始化SolrIndexSearcher(同上)
可以看到如果indexpartion没有传递默认是default,开辟空的内存,这时没有索引被读取。当有indexpartion时会读取indexpartion目录下的indexLinks。
比如/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr/data/20110111/indexLinks。indexLinks会记录索引存放的路径,然后分别加载,最后通过MultiReader实现。
索引测试路径:
local_path:/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr/data/20110111
hdfs_path:/group/tbdp-etao-adhoc/p4padhoc/tabletest/index/20110111/part-00001
最后补充solr容器的关闭,下次分析
"Shutdown" prio=10 tid=0x0000000040e97000 nid=0x7b1d at breakpoint[0x00007f7af4772000]
java.lang.Thread.State: RUNNABLE
at org.apache.solr.core.SolrCore.close(SolrCore.java:473)
at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:202)
- locked <0x00000000e4288b70> (a java.util.LinkedHashMap)
at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:104)
at org.mortbay.jetty.servlet.FilterHolder.destroyInstance(FilterHolder.java:127)
at org.mortbay.jetty.servlet.FilterHolder.doStop(FilterHolder.java:107)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000000e4280868> (a java.lang.Object)
at org.mortbay.jetty.servlet.ServletHandler.doStop(ServletHandler.java:176)
- locked <0x00000000e4280768> (a org.mortbay.jetty.servlet.ServletHandler)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000000e42807d8> (a java.lang.Object)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at org.mortbay.jetty.servlet.SessionHandler.doStop(SessionHandler.java:125)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000000e4280758> (a java.lang.Object)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at org.mortbay.jetty.handler.ContextHandler.doStop(ContextHandler.java:592)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000000e4280708> (a java.lang.Object)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at org.mortbay.jetty.Server.doStop(Server.java:283)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000000e4280658> (a java.lang.Object)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Server.java:561)
补充SolrCore初始化的stacktrace,下次分析
"2010005445@qtp-385415131-0" daemon prio=10 tid=0x0000000041df3800 nid=0x440c at breakpoint[0x00007f2686060000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:278)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:85)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:808)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:509)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:369)
at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:110)
at org.apache.lucene.search.spell.SpellChecker.createSearcher(SpellChecker.java:752)
at org.apache.lucene.search.spell.SpellChecker.swapSearcher(SpellChecker.java:729)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:174)
- locked <0x00000000e25f3818> (a java.lang.Object)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
at org.apache.solr.spelling.IndexBasedSpellChecker.init(IndexBasedSpellChecker.java:55)
at org.apache.solr.handler.component.SpellCheckComponent.inform(SpellCheckComponent.java:598)
at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:523)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:385)
at org.apache.solr.core.CoreContainer.createCore(CoreContainer.java:180)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:151)
- locked <0x00000000e2e23bf0> (a java.util.LinkedHashMap)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:148)