As Solr’s design, sentences needs to be break down(tokenized) to words for index & query. Solr introduce tokenizer and filter to accomplish this target. The work flow in solr can be simplied as follows:
- Sentences are break down to words at first by Tokenizers
- Then, some transformation is done to these words, transformations may be to lowercase/ filter out un-important words, by Filters
The combination of series of tokenizer & filters are called Analyzer