增量导入mysql数据到solr更新索引

转载自 https://blog.csdn.net/csdn_avatar_2019/article/details/89405342

1、准备

 

-- 增量测试数据
INSERT INTO meta_solr VALUES (8, 'James', 0, '美国洛杉矶湖人队', default, default);
INSERT INTO meta_solr VALUES (9, 'McGrady', 0, '美国火箭丰田球场前排', default, default);

2、修改solr-8.1.1/server/solr/meta_db/conf/data-config.xml文件

 

<?xml version="1.0" encoding="UTF-8"?>
<dataConfig>
    <dataSource name="data_source"
                type="JdbcDataSource"
                driver="com.mysql.cj.jdbc.Driver"
                url="jdbc:mysql://192.168.3.10:3306/meta_db"
                user="root"
                password="root"
                batchSize="-1"/>
    <document name="meta_solr_doc">
        <!-- 全量导入数据配置 -->
		<!--
        <entity name="meta_solr" 
				dataSource="data_source"
				pk="id" 
				query="select id, name, sex, address, DATE_FORMAT(update_time, '%Y-%m-%d %H:%i:%S') as updateTime, deleted from meta_solr">
			<field column="id" name="id"/>　　
			<field column="name" name="name"/>
			<field column="sex" name="sex"/>
			<field column="address" name="address"/>　
			<field column="updateTime" name="updateTime"/>
			<field column="deleted" name="deleted"/>
		</entity>
		-->
		<!-- 增量导入数据配置 -->
        <!-- 
            transformer 格式转化：HTMLStripTransformer 索引中忽略HTML标签
            query：查询数据库表符合记录数据
            deltaQuery：增量索引查询主键ID  注意这个只能返回ID字段   
            deltaImportQuery：增量索引查询导入的数据 
            deletedPkQuery：增量索引删除主键ID查询 注意这个只能返回ID字段 deletedPkQuery="select id from meta_solr where 1=1 and isDeleted=1"
        -->
        <entity name="meta_solr" 
                dataSource="data_source"
                pk="id" 
                query="select id, name, sex, address, DATE_FORMAT(update_time, '%Y-%m-%d %H:%i:%S') as updateTime, deleted from meta_solr"
                deltaImportQuery="select * from meta_solr where id='${dih.delta.id}'"
                deltaQuery="select id from meta_solr where update_time > '${dataimporter.last_index_time}'"> 
            <field column="id" name="id"/>　　
            <field column="name" name="name"/>
            <field column="sex" name="sex"/>
            <field column="address" name="address"/>　
            <field column="update_time" name="updateTime"/>
            <field column="deleted" name="deleted"/>
        </entity>
    </document>
</dataConfig>

3、更新字段配置，重启solr

<field name="updateTime" type="pdate" indexed="true" stored="true"/>

 

4、导入数据，这里要选delta-import，只勾选commit

5、数据库插入增量数据，再次导入

 

6、增量导入服务

https://github.com/xiekong/solr-data-import-scheduler