这段时间集成HBase,需要为HBase建立二级索引,方便数据的查询使用,Solr权威指南上面有Hbase与Solr的集成章节,照着书上以及网上的说明折腾了很近才配置成功,HBase Indexer已经有1年多没有更新了,集成最新的HBase1.2.6,solr7.2.1有很多相关的接口都发送了变化
1.下载HBaseIndexer 项目:
官网:http://ngdata.github.io/hbase-indexer/
github:https://github.com/NGDATA/hbase-indexer
wiki(配置说明):https://github.com/NGDATA/hbase-indexer/wiki
sep tools cli说明(状态监控):https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep/hbase-sep-tools
2.下载完成后修改根目录的pom.xml
配置版本调整到我们需要的hbase及solr版本,version.solr.mapreduce及httpclient.core 是自己添加的版本号,需要修改pom内相应的位置,因为这两个包跟主jar的版本不同步:
<properties>
<version.solr>7.2.1</version.solr>
<version.solr.mapreduce>6.5.1</version.solr.mapreduce>
<version.guava>12.0.1</version.guava>
<version.joda-time>1.6</version.joda-time>
<version.slf4j>1.7.7</version.slf4j>
<version.hbase>1.2.6</version.hbase>
<version.hadoop>2.7.4</version.hadoop>
<version.zookeeper>3.4.6</version.zookeeper>
<version.jackson>1.9.13</version.jackson>
<!-- version.httpclient>4.3</version.httpclient -->
<version.httpclient>4.5.3</version.httpclient>
<version.httpclient.core>4.4.6</version.httpclient.core>
<!-- <version.kite>0.13.0</version.kite> -->
<version.kite>0.15.0</version.kite>
<version.jersey>1.17</version.jersey>
<version.surefire.plugin>2.19.1</version.surefire.plugin>
<version.failsafe.plugin>${version.surefire.plugin}</version.failsafe.plugin>
<!-- Tells maven plugins what file encoding to use -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
添加plugins方便提取jar:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<outputDirectory>${project.build.directory}</outputDirectory>
<excludeTransitive>false</excludeTransitive>
</configuration>
</plugin>
3.将项目导入到eclipse(处理版本升级后带来的编译问题)
需要注意的是:pom.xml内有几个插件eclipse的m2eclipse插件不兼容,需要ctrl+i ignore掉:否则会提示Lifecycle问题
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<versionRange>1.6</versionRange>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<versionRange>2.8</versionRange>
如果在eclipse下面要运行项目的话,需要自己把这几个插件做的逻辑手动做一下:
(1).替换hbase-indexer-default.xml内的版本号
(2).生成hbase-indexer-common\src\main\java\com\ngdata\hbaseindexer\package-info.java 文件:
@VersionAnnotation(version="1.6-SNAPSHOT", revision="Unknown",
user="root", date="Thu Feb 1 21:15:10 CST 2018", url="file:///E:/work/windtrend/hbase-indexer/hbase-indexer-common")
package com.ngdata.hbaseindexer;
export HBASE_INDEXER_HEAPSIZE=1024
export HBASE_INDEXER_LOG_DIR=$HBASE_INDEXER_HOME/logs
export HBASE_INDEXER_PID_DIR=$HBASE_INDEXER_HOME/pid
export HBASE_INDEXER_CLI_ZK=master,slave1,slave2,slave3
其它配置可根据情况进行修改,如远程调试:
export HBASE_INDEXER_OPTS="$HBASE_INDEXER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8075"
<?xml version="1.0"?>
<indexer table="t_order" unique-key-field="id">
<field name="number" value="f:number" type="string"/>
<field name="source" value="f:source" type="int"/>
<field name="tenantId" value="f:tenantId" type="int"/>
<field name="userId" value="f:userId" type="int"/>
<field name="storeId" value="f:storeId" type="int"/>
<field name="storeName" value="f:storeName" type="string"/>
<field name="storeNumber" value="f:storeNumber" type="string"/>
<field name="userName" value="f:userName" type="string"/>
</indexer>
(5).新建order-schema.xml (找一个solr内的默认 schema修改即可)
<field name="number" type="string" indexed="true" stored="true"/>
<field name="source" type="int" indexed="true" stored="true"/>
<field name="tenantId" type="int" indexed="true" stored="true"/>
<field name="userId" type="int" indexed="true" stored="true"/>
<field name="storeId" type="int" indexed="true" stored="true"/>
<field name="storeName" type="string" indexed="true" stored="true"/>
<field name="storeNumber" type="string" indexed="true" stored="true"/>
<field name="userName" type="string" indexed="true" stored="true"/>
6.配置solr(solr必须是cloud模式,看了代码也可以是classic模式,但是classic模式内配置的参数全是solr.shard.xxx)
(1).配置solr cloud
(2).创建collection order
bin/solr create -c order
(3).上传schema:
./solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd putfile /configs/order/managed-schema $HBASE_INDEXER_HOME/index/order-schema.xml
注:zkcli是solr提供的脚本
(4)配置solrconfig 软提交:
./solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd getfile /configs/order/solrconfig.xml $HBASE_INDEXER_HOME/index/solrconfig.xml
修改配置并上传:
<autoSoftCommit>
<maxTime>5000</maxTime>
<maxDocs>100</maxDocs>
</autoSoftCommit>
(5) reload collection
7.配置hbase:
(1).拷贝jar包到hbase lib目录
cp ./lib/hbase-sep* $HBASE_HOME/lib/
<property>
<name>hbase.replication</name>
<value>true</value>
</property>
<property>
<name>replication.source.ratio</name>
<value>1.0</value>
</property>
<property>
<name>replication.source.nb.capacity</name>
<value>1000</value>
</property>
<property>
<name>replication.replicationsource.implementation</name>
<value>com.ngdata.sep.impl.SepReplicationSource</value>
</property>
(3).复制到各节点,并重启hbase
./bin/hbase-indexer add-indexer \
--name orderindexer \
--indexer-conf index/order-indexer.xml \
--cp solr.zk=master:9983 \
--cp solr.collection=order
(3) 在hbase 内创建表
$ hbase shell
hbase> create 't_order', { NAME => 'f', REPLICATION_SCOPE => '1' }
hbase> put 't_order', 'row1', 'f:userId', 10000
hbase> put 't_order', 'row2', 'f:userId', 10000
(4),进入solr查询就能看到结果
备注:
目前还没办法在windows环境下使用server,windows下面没有办法接受到SepEvent,调试的话需要在linux下面或者开启远程debug,hbase-indexer-evn.sh 内有参数可以开启;
几个核心处理类:
事件入口:
com.ngdata.sep.impl.SepEventExecutor.scheduleEventBatch(int partition, List<SepEvent> events)
数据处理入口:
com.ngdata.hbaseindexer.indexer.Indexer$RowBasedIndexer
Indexer com.ngdata.hbaseindexer.mr.HBaseIndexerMapper.createIndexer(String indexName, Context context, IndexerConf indexerConf, String tableName, ResultToSolrMapper mapper, Map<String, String> indexConnectionParams) throws IOException, SharderException
参考:
官网:http://ngdata.github.io/hbase-indexer/
github:https://github.com/NGDATA/hbase-indexer
wiki(配置说明):https://github.com/NGDATA/hbase-indexer/wiki
sep tools cli说明(状态监控):https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep/hbase-sep-tools
http://blog.csdn.net/d6619309/article/details/51500368