Lucene索引中有几个最基础的概念,索引(index),文档(document),域(field),和项(或者译为语词term)
其中Index为Document的序列
Document为Field的序列
Field为Term的序列
Term就是一个子串
存在于不同的Field中的同一个子串被认为是不同的Term.因此Term实际上是用一对子串表示的,第一个子串为Field的name,第二个为Field中的子串.既然Term这么重要,我们先来认识一下Term.
认识Term
最好的方法就是看其源码表示
public final class Term implements Comparable, java.io.Serializable {
String field;
String text;
public Term(String fld, String txt) {this(fld, txt, true);}
public final String field() { return field; }
public final String text() { return text; }
//overwrite equals()
public final boolean equals(Object o) { }
//overwrite hashCode()
public final int hashCode() {return field.hashCode() + text.hashCode();
}
public int compareTo(Object other) {return compareTo((Term)other);}
public final int compareTo(Term other)
final void set(String fld, String txt) public final String toString() { return field + ":" + text; }
private void readObject(java.io.ObjectInputStream in)
}
从代码中我们可以大体看出Tern其实是一个二元组
<FieldName,text>
倒排索引
为了使得基于项的搜索更有效率，索引中项是静态存储的。Lucene的索引属于索引方式中的倒排索引，因为对于一个项这种索引可以列出包含它的文档。这刚好是文档与项自然联系的倒置。
Field的类型
Lucene中，Field的文本可能以逐字的非倒排的方式存储在索引中。而倒排过的Field称为被索引过了。Field也可能同时被存储和被索引。Field的文本可能被分解许多Term而被索引，或者就被用作一个Term而被索引。大多数的Field是被分解过的，但是有些时候某些标识符域被当做一个Term索引是很有用的。
Index包中的每个类解析
CompoundFileReader
提供读取.cfs文件的方法
.
CompoundFileWriter
用来构建.cfs文件,从Lucene1.4开始,会将下面要提到的各类文件,譬如.tii,.tis等合并成一个.cfs文件
!
其结构如下
Compound (.cfs) --> FileCount, <DataOffset, FileName>FileCount, FileDataFileCount
FileCount --> VInt
DataOffset --> Long
FileName --> String
FileData --> raw file data
DocumentWriter
构建.frq,.prx,.f文件
| 1．FreqFile (.frq) --> <TermFreqs, SkipData>TermCount
TermFreqs --> <TermFreq>DocFreq
TermFreq --> DocDelta, Freq?
SkipData --> <SkipDatum>DocFreq/SkipInterval
SkipDatum --> DocSkip,FreqSkip,ProxSkip
DocDelta,Freq,DocSkip,FreqSkip,ProxSkip --> VInt
|
|
|
| 2．The .prx file contains the lists of positions that each term occurs at within documents.
ProxFile (.prx) --> <TermPositions>TermCount
TermPositions --> <Positions>DocFreq
Positions --> <PositionDelta>Freq
PositionDelta --> VInt
|
3．There's a norm file for each indexed field with a byte for each document. The .f[0-9]* file contains, for each document, a byte that encodes a value that is multiplied into the score for hits on that field:
Norms (.f[0-9]*) --> <Byte>SegSize
Each byte encodes a floating point value. Bits 0-2 contain the 3-bit mantissa, and bits 3-8 contain the 5-bit exponent.
These are converted to an IEEE single float value as follows:
1. If the byte is zero, use a zero float.
2. Otherwise, set the sign bit of the float to zero;
3. add 48 to the exponent and use this as the float's exponent;
4. map the mantissa to the high-order 3 bits of the float's mantissa; and
5. set the low-order 21 bits of the float's mantissa to zero.
FieldInfo
里边有Field的部分信息,是一个四元组<name,isIndexed,num,
storeTermVector>
FieldInfos
此类用来描述Document的fields是否被索引.每个Segment有一个单独的FieldInfo 文件.对于多线程,此类的对象为线程安全的.但是某一时刻,只允许一个线程添加document.别的reader和writer不允许进入.此类维护两个容器ArrayList和HashMap,这两个容器都不是synchronized,何言线程安全,不解
??
观察write函数可知 .fnm文件的构成为
FieldInfos (.fnm) --> FieldsCount, <FieldName, FieldBits>FieldsCount
FieldsCount --> VInt
FieldName --> String
FieldBits --> Byte
FieldReader
用来读取.fdx文件和.fdt文件
FieldWriter
此类创建两个文件.fdx和.fdt文件
FieldIndex(.fdx)对于每一个Document,里面都含有一个指向Field的指针(其实是整数
)
<FieldValuesPosition>SegSize
FieldValuesPosition --> Uint64
则第n个document的Field pointer为
n*8
FieldData(.fdt)里面包含了每一个文档包含的存储的field信息.内容如下
:
<DocFieldData>SegSize
DocFieldData --> FieldCount, <FieldNum, Bits, Value>FieldCount
FieldCount --> VInt
FieldNum --> VInt
Lucene <= 1.4:
Bits --> Byte
Value --> String
Only the low-order bit of Bits is used. It is one for tokenized fields, and zero for non-tokenized fields.
FilterIndexReader
扩展自IndexReader,提供了具体的方法
.
IndexReader
为abstract class!用来读取建完索引的Directory,并可以返回各种信息,譬如Term,TermPosition等等
.
IndexWriter
IndexWriter用来创建和维护索引。
IndexWriter构造函数中的第三个参数决定一个新的索引是否被创建，或者一个存在的索引是否开放给欲新加入的新的
document
通过addDocument()0函数加入新的documents，当添加完document之后，调用close()函数
如果一个Index没有document需要加入并且需要优化查询性能。则在索引close()之前，调用optimize()函数进行优化。
Deleteable文件结构：
A file named "deletable" contains the names of files that are no longer used by the index, but which could not be deleted. This is only used on Win32, where a file may not be deleted while it is still open. On other platforms the file contains only null bytes.
Deletable --> DeletableCount, <DelableName>DeletableCount
DeletableCount --> UInt32
DeletableName --> String
MultipleTermPositions
专门用于search包中的
PhrasePrefixQuery
MultiReader
扩展自IndexReader，用来读取多个索引！添加他们的内容
SegmentInfo
一些关于Segment的信息,是一个三元组
<segmentname,docCount,dir>
SegmentInfos
扩展自Vector,就是一个向量组,其中任意成员为SegmentInfo!用来构建segments文件,每个Index有且只有一个这样的文件,此类提供了read和write的方法
.
其内容如下
:
Segments --> Format, Version, NameCounter, SegCount, <SegName, SegSize>SegCount
Format, NameCounter, SegCount, SegSize --> UInt32
Version --> UInt64
SegName --> String
Format is -1 in Lucene 1.4.
Version counts how often the index has been changed by adding or deleting documents.
NameCounter is used to generate names for new segment files.
SegName is the name of the segment, and is used as the file name prefix for all of the files that compose the segment's index.
SegSize is the number of documents contained in the segment index.
SegmentMergeInfo
用来记录segment合并信息
.
SegmentMergeQueue
扩展自PriorityQueue(按升序排列
)
SegmentMerger
此类合并多个Segment为一个Segment，被IndexWriter.addIndexes()创建此类对象
如果compoundFile为True即可以合并了,创建.cfs文件，并且把其余的几乎所有文件全部合并到.cfs文件中！
SegmentReader
扩展自IndexReader，提供了很多读取Index的方法
SegmentTermDocs
扩展自
TermDocs
SegmentTermEnum
扩展自
TermEnum
SegmentTermPositions
扩展自
TermPositions
SegmentTermVector
扩展自
TermFreqVector
Term
Term是一个<fieldName,text>对.而Field由于分多种,但是至少都含有<fieldName,fieldValue>这样二者就可以建立关联了.Term是一个搜索单元.Term的text都是诸如dates,email address,urls等等
.
TermDocs
TermDocs是一个Interface. TermDocs提供一个接口,用来列举<document,frequency>,以共Term使用
在<document,frequency>对中,document部分给每一个含有term的document命名.document根据其document number进行标引.frequency部分列举在每一个document中term的数量.<document,frequency>对根据document number排序
.
TermEnum
此类为抽象类,用来enumerate term.Term enumerations 由Term.compareTo()进行排序此enumeration中的每一个term都要大于所有在此enumeration之前的
term.
TermFreqVector
此Interface用来访问一个document的Field的
Term Vector
TermInfo
此类主要用来存储Term信息.其可以说为一个五元组
<Term,docFreq,freqPointer,proxPointer,skipOffset>
TermInfoReader
未细读,待读完
SegmentTermEnum
TermInfoWriter
此类用来构建(.tis)和(.tii)文件.这些构成了
term dictionary
1. The term infos, or tis file.
TermInfoFile (.tis)--> TIVersion, TermCount, IndexInterval, SkipInterval, TermInfos
TIVersion --> UInt32
TermCount --> UInt64
IndexInterval --> UInt32
SkipInterval --> UInt32
TermInfos --> <TermInfo>TermCount
TermInfo --> <Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>
Term --> <PrefixLength, Suffix, FieldNum>
Suffix --> String
PrefixLength, DocFreq, FreqDelta, ProxDelta, SkipDelta
--> VInt
This file is sorted by Term. Terms are ordered first lexicographically by the term's field name, and within that lexicographically by the term's text.
TIVersion names the version of the format of this file and is -2 in Lucene 1.4.
Term text prefixes are shared. The PrefixLength is the number of initial characters from the previous term which must be pre-pended to a term's suffix in order to form the term's text. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y".
FieldNumber determines the term's field, whose name is stored in the .fdt file.
DocFreq is the count of documents which contain the term.
FreqDelta determines the position of this term's TermFreqs within the .frq file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the file).
ProxDelta determines the position of this term's TermPositions within the .prx file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the file.
SkipDelta determines the position of this term's SkipData within the .frq file. In particular, it is the number of bytes after TermFreqs that the SkipData starts. In other words, it is the length of the TermFreq data.
2. The term info index, or .tii file.
This contains every IndexIntervalth entry from the .tis file, along with its location in the "tis" file. This is designed to be read entirely into memory and used to provide random access to the "tis" file.
The structure of this file is very similar to the .tis file, with the addition of one item per record, the IndexDelta.
TermInfoIndex (.tii)--> TIVersion, IndexTermCount, IndexInterval, SkipInterval, TermIndices
TIVersion --> UInt32
IndexTermCount --> UInt64
IndexInterval --> UInt32
SkipInterval --> UInt32
TermIndices --> <TermInfo, IndexDelta>IndexTermCount
IndexDelta --> VLong
IndexDelta determines the position of this term's TermInfo within the .tis file. In particular, it is the difference between the position of this term's entry in that file and the position of the previous term's entry.
TODO: document skipInterval information
其中IndexDelta是.tii文件,比之.tis文件多的东西
.
TermPosition
此类扩展自TermDocs,是一个Interface,用来enumerate<document,frequency,<position>*>三元组
,
以供term使用.在此三元组中document和frequency于TernDocs中的相同.postions部分列出了在一个document中,一个term每次出现的顺序位置此三元组为倒排文档的事件表表示
.
TermPositionVector
扩展自TermFreqVector.比之TermFreqVector扩展了功能,可以提供term所在的位置
TermVectorReader
用来读取.tvd,.tvf.tvx三个文件
.
TermVectorWriter
用于构建.tvd, .tvf,.tvx文件,这三个文件构成
TermVector
| 1. The Document Index or .tvx file.
This contains, for each document, a pointer to the document data in the Document (.tvd) file.
DocumentIndex (.tvx) --> TVXVersion<DocumentPosition>NumDocs
TVXVersion --> Int
DocumentPosition --> UInt64
This is used to find the position of the Document in the .tvd file.
2. The Document or .tvd file.
This contains, for each document, the number of fields, a list of the fields with term vector info and finally a list of pointers to the field information in the .tvf (Term Vector Fields) file.
Document (.tvd) --> TVDVersion<NumFields, FieldNums, FieldPositions,>NumDocs
TVDVersion --> Int
NumFields --> VInt
FieldNums --> <FieldNumDelta>NumFields
FieldNumDelta --> VInt
FieldPositions --> <FieldPosition>NumFields
FieldPosition --> VLong
The .tvd file is used to map out the fields that have term vectors stored and where the field information is in the .tvf file.
3. The Field or .tvf file.
This file contains, for each field that has a term vector stored, a list of the terms and their frequencies.
Field (.tvf) --> TVFVersion<NumTerms, NumDistinct, TermFreqs>NumFields
TVFVersion --> Int
NumTerms --> VInt
NumDistinct --> VInt -- Future Use
TermFreqs --> <TermText, TermFreq>NumTerms
TermText --> <PrefixLength, Suffix>
PrefixLength --> VInt
Suffix --> String
TermFreq --> VInt
Term text prefixes are shared. The PrefixLength is the number of initial characters from the previous term which must be pre-pended to a term's suffix in order to form the term's text. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y".
|
好的，整个Index包所有类都讲解了，下边咱们开始来编码重新审视一下！
下边来编制一个程序来结束本章的讨论。
package org.apache.lucene.index;
import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.store.*;
import org.apache.lucene.document.*;
import org.apache.lucene.demo.*;
import org.apache.lucene.search.*;
import java.io.*;
/**在使用此程序时，会尽量用到Lucene Index中的每一个类，尽量将其展示个大家
*使用的Index包中类有
*DocumentWriter(提供给用用户使用的为IndexWriter）
*FieldInfo(和FieldInfos）
* SegmentDocs（扩展自TermDocs）
*SegmentReader（扩展自IndexReader，提供给用户使用的是IndexReader）
*SegmentMerger
*segmentTermEnum(扩展自TermEnum）
*segmentTermPositions(扩展自TermPositions）
*segmentTermVector(扩展自TermFreqVector）
*/
public class TestIndexPackage
{
//用于将Document加入索引
public static void indexDocument(String segment,String fileName) throws Exception
{
//第二个参数用来控制，如果获得不了目录是否创建
Directory directory = FSDirectory.getDirectory("testIndexPackage",false);
Analyzer analyzer = new SimpleAnalyzer();
//第三个参数为每一个Field最多拥有的Token个数
DocumentWriter writer = new DocumentWriter(directory,analyzer,Similarity.getDefault(),1000);
File file = new File(fileName);
//由于使用FileDocument将file包装成了Docuement，会在document中创建三个field（path，modified，contents）
Document doc = FileDocument.Document(file);
writer.addDocument(segment,doc);
directory.close();
}
//将多个segment进行合并
public static void merge(String segment1,String segment2,String segmentMerged)throws Exception
{
Directory directory = FSDirectory.getDirectory("testIndexPackage",false);
SegmentReader segmentReader1=new SegmentReader(new SegmentInfo(segment1,1,directory));
SegmentReader segmentReader2=new SegmentReader(new SegmentInfo(segment2,1,directory));
//第三个参数为是否创建.cfs文件
SegmentMerger segmentMerger =new SegmentMerger(directory,segmentMerged,false);
segmentMerger.add(segmentReader1);
segmentMerger.add(segmentReader2);
segmentMerger.merge();
segmentMerger.closeReaders();
directory.close();
}
//将segment即Index的子索引的所有内容展示给你看。
public static void printSegment(String segment) throws Exception
{
Directory directory =FSDirectory.getDirectory("testIndexPackage",false);
SegmentReader segmentReader = new SegmentReader(new SegmentInfo(segment,1,directory));
//display documents
for(int i=0;i<segmentReader.numDocs();i++)
System.out.println(segmentReader.document(i));
TermEnum termEnum = segmentReader.terms();//此处实际为
SegmentTermEnum
//display term and term positions,termDocs
while(termEnum.next())
{
System.out.print(termEnum.term().toString2());
System.out.println(" DocumentFrequency=" + termEnum.docFreq());
TermPositions termPositions= segmentReader.termPositions(termEnum.term());
int i=0;
while(termPositions.next())
{
System.out.println((i++)+"->"+termPositions);
}
TermDocs termDocs=segmentReader.termDocs(termEnum.term());//实际为
segmentDocs
while (termDocs.next())
{
System.out.println((i++)+"->"+termDocs);
}
}
//display field info
FieldInfos fieldInfos= segmentReader.fieldInfos;
FieldInfo pathFieldInfo = fieldInfos.fieldInfo("path");
FieldInfo modifiedFieldInfo = fieldInfos.fieldInfo("modified");
FieldInfo contentsFieldInfo =fieldInfos.fieldInfo("contents");
System.out.println(pathFieldInfo);
System.out.println(modifiedFieldInfo);
System.out.println(contentsFieldInfo);
//display TermFreqVector
for(int i=0;i<segmentReader.numDocs();i++)
{
//对contents的token之后的term存于了
TermFreqVector
TermFreqVector termFreqVector=segmentReader.getTermFreqVector(i,"contents");
System.out.println(termFreqVector);
}
}
public static void main(String [] args)
{
try
{
Directory directory = FSDirectory.getDirectory("testIndexPackage",true);
directory.close();
indexDocument("segmentOne","e://lucene//test.txt");
//printSegment("segmentOne");
indexDocument("segmentTwo","e://lucene//test2.txt");
// printSegment("segmentTwo");
merge("segmentOne","segmentTwo","merge");
printSegment("merge");
}
catch(Exception e)
{
System.out.println("caught a "+e.getCause()+"/n with message:"+e.getMessage());
e.printStackTrace();
}
}
}
看看其结果如下：
Document<Text<path:e:/lucene/test.txt> Keyword<modified:0eg4e221c>>
Document<Text<path:e:/lucene/test2.txt> Keyword<modified:0eg4ee8b4>>
<Term:FieldName,text>=<contents,china> DocumentFrequency=1
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=2>
1-><docNumber,freq>=<0,1>
<Term:FieldName,text>=<contents,i> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=2 Pos=0,3>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=0>
2-><docNumber,freq>=<0,2>
3-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<contents,love> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=2 Pos=1,4>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=1>
2-><docNumber,freq>=<0,2>
3-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<contents,nankai> DocumentFrequency=1
0-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=2>
1-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<contents,tianjin> DocumentFrequency=1
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=5>
1-><docNumber,freq>=<0,1>
<Term:FieldName,text>=<modified,0eg4e221c> DocumentFrequency=1
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=0>
1-><docNumber,freq>=<0,1>
<Term:FieldName,text>=<modified,0eg4ee8b4> DocumentFrequency=1
0-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=0>
1-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<path,e> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=0>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=0>
2-><docNumber,freq>=<0,1>
3-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<path,lucene> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=1>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=1>
2-><docNumber,freq>=<0,1>
3-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<path,test> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=2>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=2>
2-><docNumber,freq>=<0,1>
3-><docNumber,freq>=<1,1>
<Term:FieldName,text>=<path,txt> DocumentFrequency=2
0-><doc,TermFrequency,Pos>:< doc=0, TermFrequency=1 Pos=3>
1-><doc,TermFrequency,Pos>:< doc=1, TermFrequency=1 Pos=3>
2-><docNumber,freq>=<0,1>
3-><docNumber,freq>=<1,1>
<fieldName,isIndexed,fieldNumber,storeTermVector>=path,true,3,false>
<fieldName,isIndexed,fieldNumber,storeTermVector>=modified,true,2,false>
<fieldName,isIndexed,fieldNumber,storeTermVector>=contents,true,1,true>
{contents: china/1, i/2, love/2, tianjin/1}
{contents: i/1, love/1, nankai/1}
认真审视其结果，你就会更加明白Lucene底层的索引结构如何。
参考资料：Lucene File Format
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
