实时查询方案
Hbase -----> Key Value Store ---> Solr ------->Web前端实时查询展示
1.Hbase 提供海量数据存储
2.Solr提供索引构建与查询
3. Key Value Store 提供自动化索引构建(从Hbase到Solr)
使用流程
前提: CDH5.3.2Solr集群搭建好,CDH5.3.2 Key-Value StoreIndexer集群搭建好
1.开启Hbase的复制功能
2. Hbase表需要开启REPLICATION复制功能
create 'table',{NAME => 'cf', REPLICATION_SCOPE => 1} #其中1表示开启replication功能,0表示不开启,默认为0 |
对于已经创建的表可以使用如下命令
disable 'table' alter 'table',{NAME => 'cf', REPLICATION_SCOPE => 1} enable 'table' |
3. 生成实体配置文件, /opt/hbase-indexer/Test是自定义路径,可以自己设置
solrctl instancedir --generate /opt/cdhsolr/waslog
4.编辑生成好的scheme.xml文件
把hbase表中需要索引的列添加到scheme.xml filed节点
5.创建collection实例并配置文件上传到zookeeper,命令
solrctl instancedir --create waslog /opt/cdhsor/waslog
6.上传到zookeeper之后,其他节点就可以从zookeeper下载配置文件。接下来创建collection,命令:
solrctl collection –create waslog -s 15 –r 2 –m 50
7 在一个目录创建一个xml文件,该文件确定solr和hbase的关系,示例如下
<?xml version="1.0" encoding="UTF-8"?>
<indexer table="pucentp_solr">
<field name="entpname" value="cf:entpname" />
<field name="address" value="cf:address" />
<field name="policyname" value="cf:policyname" />
<field name="policytel" value="cf:policytel" />
<field name="createtime" value="cf:createtime" />
<field name="updatetime" value="cf:updatetime" />
</indexer>
Table对应hbase表,fieldname对应solr里的索引字段 value是由列簇:列组成
8 在hbase-solr目录下的bin目录下执行hbase-indexer(cd /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/bin)
./hbase-indexeradd-indexer -n myindexer -c /home/test/index/mapper.xml -cpsolr.zk=master:2181,slaver1:2181,slaver2:2181/solr -cp solr.collection=waslog-z master:2181,slaver1:2181,slaver2:2181
添加拼音分词
添加 pinyin4j-2.5.0.jar lucene-analyzers-smartcn-4.10.3.jar 到/opt/cloudera/parcels/CDH/lib/solr/webapps/solr/WEB-INF/lib下。分发到其它从节点
修改schemal.xml
1. <fieldType name="text_pinyin" class="solr.TextField" positionIncrementGap="0">
2. <analyzer type="index">
3. <tokenizer class="org.apache.lucene.analysis.cn.smart.SmartChineseSentenceTokenizerFactory"/>
4. <filter class="org.apache.lucene.analysis.cn.smart.SmartChineseWordTokenFilterFactory"/>
5. <filter class="com.shentong.search.analyzers.PinyinTransformTokenFilterFactory" minTermLenght="2" />
6. <filter class="com.shentong.search.analyzers.PinyinNGramTokenFilterFactory" minGram="1" maxGram="20" />
7. </analyzer>
8.
9. <analyzer type="query">
10. <tokenizer class="org.apache.lucene.analysis.cn.smart.SmartChineseSentenceTokenizerFactory"/>
11. <filter class="org.apache.lucene.analysis.cn.smart.SmartChineseWordTokenFilterFactory"/>
12. <filter class="com.shentong.search.analyzers.PinyinTransformTokenFilterFactory" minTermLenght="2" />
13. <filter class="com.shentong.search.analyzers.PinyinNGramTokenFilterFactory" minGram="1" maxGram="20" />
14. </analyzer>
15. </fieldType>
添加smartcn分词
1. <fieldType name="text_smartcn" class="solr.TextField" positionIncrementGap="0">
2. <analyzer type="index">
3. <tokenizer class="org.apache.lucene.analysis.cn.smart.SmartChineseSentenceTokenizerFactory"/>
4. <filter class="org.apache.lucene.analysis.cn.smart.SmartChineseWordTokenFilterFactory"/>
5. </analyzer>
6. <analyzer type="query">
7. <tokenizer class="org.apache.lucene.analysis.cn.smart.SmartChineseSentenceTokenizerFactory"/>
8. <filter class="org.apache.lucene.analysis.cn.smart.SmartChineseWordTokenFilterFactory"/>
9. </analyzer>
10. </fieldType>
重启solr集群