- 一个曾实现的简单思路:
自动提示功能,以前的有一种实现思路就是在数据库里建一张表,其主要字段有:
keyword-检索关键字;kcount-检索次数;dissect_word-对检索关键字分词后的结果;kdate:检索时间
由于用户输入的检索关键字可能很乱,又可能很杂,所以想到通过分词器把检索关键字进行分词处理,若数据库中在dissect_word中找到含有相同的值则认为是检索相同的关键字,kcount+1.
当用户检索“lucene”的时候就根据keyword like "lucene%" order by kcount desc 来展示给前台。
虽然看起来像是那么回事了,但是没有专业性。今天研究了一下solr的自动提示组件TermsComponent类文件。
- solr中TermsComponent源代码:
为了便于大家理解可以先执行下面的测试代码:
FSDirectory directory
= FSDirectory.open(
new File(
"D://DATAMANAGER//INDEX//SYS_3000"));
IndexReader r = IndexReader.open(directory);
//TermEnum te = r.terms(); //测试1
TermEnum te = r.terms( new Term( "content", "防")); //测试2
while(te.next()){
Term t = te.term();
if(t.field().equals( "content")){
System.out.print(t.field() + "=" +t.text() + ":" +te.docFreq() + ";");
}
}
IndexReader r = IndexReader.open(directory);
//TermEnum te = r.terms(); //测试1
TermEnum te = r.terms( new Term( "content", "防")); //测试2
while(te.next()){
Term t = te.term();
if(t.field().equals( "content")){
System.out.print(t.field() + "=" +t.text() + ":" +te.docFreq() + ";");
}
}
分别执行一下“测试1”和“测试2”,查看一下打印输出结果。其中“D://DATAMANAGER//INDEX//SYS_3000”换成你的索引文件目录;“content”为Field域字段;“防”为content域字段的一个term的text值,你可以根据你自己的环境相应替换这些值。
TermsComponent主要理解了一下它的思路,在代码上加了些注释。
public static final int UNLIMITED_MAX_COUNT = -1;
//例如,请求参数串:terms=true&terms.fl=name&terms.lower=py&terms.prefix=py&terms.lower.incl=false&indent=true&wt=json
public void process(ResponseBuilder rb) throws IOException {
SolrParams params = rb.req.getParams();
if (params.getBool(TermsParams.TERMS, false)) {//判断请求传来的参数terms=true
String lowerStr = params.get(TermsParams.TERMS_LOWER, null);//开始Term terms.lower=py
String[] fields = params.getParams(TermsParams.TERMS_FIELD); //在哪个域 terms.fl=name
if (fields != null && fields.length > 0) {
NamedList terms = new NamedList();
rb.rsp.add("terms", terms);
int limit = params.getInt(TermsParams.TERMS_LIMIT, 10); //返回个数
if (limit < 0) {
limit = Integer.MAX_VALUE;
}
String upperStr = params.get(TermsParams.TERMS_UPPER); //截止Term
boolean upperIncl = params.getBool(TermsParams.TERMS_UPPER_INCLUSIVE, false);
boolean lowerIncl = params.getBool(TermsParams.TERMS_LOWER_INCLUSIVE, true);
boolean sort = !TermsParams.TERMS_SORT_INDEX.equals(
params.get(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_COUNT)); //按索引、数量排序
int freqmin = params.getInt(TermsParams.TERMS_MINCOUNT, 1); // initialize freqmin 限制最小频率
int freqmax = params.getInt(TermsParams.TERMS_MAXCOUNT, UNLIMITED_MAX_COUNT); // initialize freqmax 限制最大频率,UNLIMITED_MAX_COUNT=-1代表不限制
if (freqmax<0) {
freqmax = Integer.MAX_VALUE;
}
String prefix = params.get(TermsParams.TERMS_PREFIX_STR); //前缀
boolean raw = params.getBool(TermsParams.TERMS_RAW, false); //是否做类型转换
for (int j = 0; j < fields.length; j++) {
String field = StringHelper.intern(fields[j]);
FieldType ft = raw ? null : rb.req.getSchema().getFieldTypeNoEx(field);
if (ft==null) ft = new StrField();
// If no lower bound was specified, use the prefix
String lower = lowerStr==null ? prefix : (raw ? lowerStr : ft.toInternal(lowerStr));
if (lower == null) lower="";
String upper = upperStr==null ? null : (raw ? upperStr : ft.toInternal(upperStr));
Term lowerTerm = new Term(field, lower);
Term upperTerm = upper==null ? null : new Term(field, upper);
TermEnum termEnum = rb.req.getSearcher().getReader().terms(lowerTerm); //this will be positioned ready to go
int i = 0;
BoundedTreeSet<CountPair<String, Integer>> queue = (sort ? new BoundedTreeSet<CountPair<String, Integer>>(limit) : null);
NamedList fieldTerms = new NamedList();
terms.add(field, fieldTerms);
Term lowerTestTerm = termEnum.term();
//Only advance the enum if we are excluding the lower bound and the lower Term actually matches
if (lowerTestTerm!=null && lowerIncl == false && lowerTestTerm.field() == field // intern'd comparison
&& lowerTestTerm.text().equals(lower)) {
termEnum.next();
}
//IndexReader.terms()返回的Term默认是根据text的字符前缀相同排列在一起的,因此当没有找到匹配的,则后面也不会再有匹配的Term
while (i<limit || sort) {
Term theTerm = termEnum.term();
// check for a different field, or the end of the index.
if (theTerm==null || field != theTerm.field()) // intern'd comparison
break;
String indexedText = theTerm.text();
// stop if the prefix doesn't match
if (prefix != null && !indexedText.startsWith(prefix)) break;
if (upperTerm != null) {
int upperCmp = theTerm.compareTo(upperTerm);
// if we are past the upper term, or equal to it (when don't include upper) then stop.
if (upperCmp>0 || (upperCmp==0 && !upperIncl)) break;
}
// This is a good term in the range. Check if mincount/maxcount conditions are satisfied.
//实现自动提示的核心代码如下:
int docFreq = termEnum.docFreq();
if (docFreq >= freqmin && docFreq <= freqmax) { //判断当前Term频率是否在有效范围内
// add the term to the list
String label = raw ? indexedText : ft.indexedToReadable(indexedText);
if (sort) {//排序(在这个队列方法里会根据频率排序,当队列个数大于初始化空间大小则自动移除最后一个元素)(默认则根据频率排序)
queue.add(new CountPair<String, Integer>(label, docFreq));
} else {//不需要排序则直接取出前缀相匹配的Term值与频率,最多取出limit个(不会考虑频率的排序)
fieldTerms.add(label, docFreq);
i++;
}
//下面给出获取一个索引文件content域字段的Term打印出来的结果,相信有助于大家理解上面的核心代码:
//content=防旱:1;content=防汛:1;content=防火:1;content=防风:1;content=阳春:1;content=陈:63;content=陈少:1;content=陈总:2;content=陈振:121;content=陈旗:16;content=陈根:6;content=限:1;content=集团:300;
//其中格式为:Field名=Term的text值:Term的频率值,从上面的输出结果中可看出其前缀相同的会紧跟在一块,但是默认是不会根据Term频率排序,也没体现出按字母顺序的规则。
}
termEnum.next();
}
termEnum.close();
if (sort) {//若排序
for (CountPair<String, Integer> item : queue) {
if (i < limit) {
fieldTerms.add(item.key, item.val);
i++;
} else {
break;
}
}
}
}
} else {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No terms.fl parameter specified");
}
}
}
//例如,请求参数串:terms=true&terms.fl=name&terms.lower=py&terms.prefix=py&terms.lower.incl=false&indent=true&wt=json
public void process(ResponseBuilder rb) throws IOException {
SolrParams params = rb.req.getParams();
if (params.getBool(TermsParams.TERMS, false)) {//判断请求传来的参数terms=true
String lowerStr = params.get(TermsParams.TERMS_LOWER, null);//开始Term terms.lower=py
String[] fields = params.getParams(TermsParams.TERMS_FIELD); //在哪个域 terms.fl=name
if (fields != null && fields.length > 0) {
NamedList terms = new NamedList();
rb.rsp.add("terms", terms);
int limit = params.getInt(TermsParams.TERMS_LIMIT, 10); //返回个数
if (limit < 0) {
limit = Integer.MAX_VALUE;
}
String upperStr = params.get(TermsParams.TERMS_UPPER); //截止Term
boolean upperIncl = params.getBool(TermsParams.TERMS_UPPER_INCLUSIVE, false);
boolean lowerIncl = params.getBool(TermsParams.TERMS_LOWER_INCLUSIVE, true);
boolean sort = !TermsParams.TERMS_SORT_INDEX.equals(
params.get(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_COUNT)); //按索引、数量排序
int freqmin = params.getInt(TermsParams.TERMS_MINCOUNT, 1); // initialize freqmin 限制最小频率
int freqmax = params.getInt(TermsParams.TERMS_MAXCOUNT, UNLIMITED_MAX_COUNT); // initialize freqmax 限制最大频率,UNLIMITED_MAX_COUNT=-1代表不限制
if (freqmax<0) {
freqmax = Integer.MAX_VALUE;
}
String prefix = params.get(TermsParams.TERMS_PREFIX_STR); //前缀
boolean raw = params.getBool(TermsParams.TERMS_RAW, false); //是否做类型转换
for (int j = 0; j < fields.length; j++) {
String field = StringHelper.intern(fields[j]);
FieldType ft = raw ? null : rb.req.getSchema().getFieldTypeNoEx(field);
if (ft==null) ft = new StrField();
// If no lower bound was specified, use the prefix
String lower = lowerStr==null ? prefix : (raw ? lowerStr : ft.toInternal(lowerStr));
if (lower == null) lower="";
String upper = upperStr==null ? null : (raw ? upperStr : ft.toInternal(upperStr));
Term lowerTerm = new Term(field, lower);
Term upperTerm = upper==null ? null : new Term(field, upper);
TermEnum termEnum = rb.req.getSearcher().getReader().terms(lowerTerm); //this will be positioned ready to go
int i = 0;
BoundedTreeSet<CountPair<String, Integer>> queue = (sort ? new BoundedTreeSet<CountPair<String, Integer>>(limit) : null);
NamedList fieldTerms = new NamedList();
terms.add(field, fieldTerms);
Term lowerTestTerm = termEnum.term();
//Only advance the enum if we are excluding the lower bound and the lower Term actually matches
if (lowerTestTerm!=null && lowerIncl == false && lowerTestTerm.field() == field // intern'd comparison
&& lowerTestTerm.text().equals(lower)) {
termEnum.next();
}
//IndexReader.terms()返回的Term默认是根据text的字符前缀相同排列在一起的,因此当没有找到匹配的,则后面也不会再有匹配的Term
while (i<limit || sort) {
Term theTerm = termEnum.term();
// check for a different field, or the end of the index.
if (theTerm==null || field != theTerm.field()) // intern'd comparison
break;
String indexedText = theTerm.text();
// stop if the prefix doesn't match
if (prefix != null && !indexedText.startsWith(prefix)) break;
if (upperTerm != null) {
int upperCmp = theTerm.compareTo(upperTerm);
// if we are past the upper term, or equal to it (when don't include upper) then stop.
if (upperCmp>0 || (upperCmp==0 && !upperIncl)) break;
}
// This is a good term in the range. Check if mincount/maxcount conditions are satisfied.
//实现自动提示的核心代码如下:
int docFreq = termEnum.docFreq();
if (docFreq >= freqmin && docFreq <= freqmax) { //判断当前Term频率是否在有效范围内
// add the term to the list
String label = raw ? indexedText : ft.indexedToReadable(indexedText);
if (sort) {//排序(在这个队列方法里会根据频率排序,当队列个数大于初始化空间大小则自动移除最后一个元素)(默认则根据频率排序)
queue.add(new CountPair<String, Integer>(label, docFreq));
} else {//不需要排序则直接取出前缀相匹配的Term值与频率,最多取出limit个(不会考虑频率的排序)
fieldTerms.add(label, docFreq);
i++;
}
//下面给出获取一个索引文件content域字段的Term打印出来的结果,相信有助于大家理解上面的核心代码:
//content=防旱:1;content=防汛:1;content=防火:1;content=防风:1;content=阳春:1;content=陈:63;content=陈少:1;content=陈总:2;content=陈振:121;content=陈旗:16;content=陈根:6;content=限:1;content=集团:300;
//其中格式为:Field名=Term的text值:Term的频率值,从上面的输出结果中可看出其前缀相同的会紧跟在一块,但是默认是不会根据Term频率排序,也没体现出按字母顺序的规则。
}
termEnum.next();
}
termEnum.close();
if (sort) {//若排序
for (CountPair<String, Integer> item : queue) {
if (i < limit) {
fieldTerms.add(item.key, item.val);
i++;
} else {
break;
}
}
}
}
} else {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No terms.fl parameter specified");
}
}
}