Solr:Text analysis

 
 

 

 

 

 

 

 

------------------------------------------------------------------------------------------------------------------------------------

Defining a custom field type for microblog text

 

 

 

------------------------------------------------------------------------------------------------------------------------------------

Advanced text analysis

 
 
 

How do you select the right text analyzer during indexing? Assuming you want to index all your documents regardless of language in the same index, a simple solution would be to use a unique field for each language. Suppose we want to index French tweets in our microblog search application. We could define the following field:

<field name="text_fr" type="text_microblog_fr"
indexed="true" stored="true" />

 

-------------------------------------------------------------------------------------------------------------------------------------

Integrate jcseg with solr to deal with chinese tokenizer

1.cp  jcseg-core-1.9.5.jar and   jcseg-solr-1.9.5.jar  to solr-4.7.0/example/solr-webapp/webapp/WEB-INF/lib/

2. cp lexicon dir to solr-4.7.0/example/solr-webapp/webapp/WEB-INF/lib/

3. alter schema.xml add fildtype

<fieldtype name="textComplex" class="solr.TextField">
   <analyzer>
       <tokenizer class="org.lionsoul.jcseg.solr.JcsegTokenizerFactory" mode="complex"/>
       <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ch.txt"/>
    </analyzer>
</fieldtype>