20161216-solr cloud 集群数据导入(dataimport)笔记

最新推荐文章于 2023-03-22 17:33:23 发布
最新推荐文章于 2023-03-22 17:33:23 发布
阅读量2.5k 收藏 2
点赞数 1
分类专栏： solr 6.3.0 zookeeper 3.4.9 集群
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/yehuang_0801/article/details/53692989
版权

在前两篇blog的基础上，进行数据的导入，连接mysql数据库。


在192.168.244.201(Master)机器上，复制cloud_conf重命名col1216_cloud_conf，修改solrconfig.xml和data-config.xml，复制重命名一个delta-data-config.xml。

solrconfig.xml加上内容如下：

  <!-- Legacy config for the admin interface -->
  <admin>
    <defaultQuery>*:*</defaultQuery>
  </admin>
 <unlockOnStartup>true</unlockOnStartup>

 <requestHandler name="/update" class="solr.UpdateRequestHandler">
	<lst name="defaults">
		<str name="update.chain">dispup</str>
	</lst>
 </requestHandler>

 <updateRequestProcessorChain name="dispup">
	<processor class="solr.UUIDUpdateProcessorFactory">
		<str name="fieldName">id</str>
	</processor>
	<processor class="solr.LogUpdateProcessorFactory" />
	<processor class="solr.DistributedUpdateProcessorFactory" />
	<processor class="solr.RunUpdateProcessorFactory" />
 </updateRequestProcessorChain>

 <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
	<lst name="defaults">
		<str name="config">data-config.xml</str>
	</lst>
 </requestHandler>

<!--
 <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
	<lst name="defaults">
		<str name="config">delta-data-config.xml</str>
		<str name="update.chain">dispup</str>
	</lst>
 </requestHandler>
-->

data-config.xml 

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
  -->

<dataConfig>
 <dataSource name="gic_read" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
  url="jdbc:mysql://172.30.2.28:18806/gic?useUnicode=true&characterEncoding=utf-8"
      user="xxx" password="xxxxx" batchSize="-1" />

<document>
	<entity name="platformgoods" dataSource="gic_read" pk="id" query="SELECT
			CONCAT('', g.goodscode) AS id,
			g.goodscode AS goodsId,
			p.id AS productId,
			p.goods_name AS goodsName,
			IFNULL(p.logourl, '') AS goodsLogo,
			p.weixin_shop_price AS goodsPrice,
			g.goods_salenum AS goodsSales,
			p.goods_inventory AS goodsInventory,
			g.`status` AS `status`,
			g.shopid AS shopId,
			'1' AS goodsType,
			vc.fid AS fCateId,
			vc.fcatalog AS fCateName,
			vc.sid AS sCateId,
			vc.scatalog AS sCateName,
			vc.tid AS tCateId,
			vc.tcatalog AS tCateName,
			p.goods_brand_id AS brandId,
			p.`name` AS brandName,
			area.pid AS provId,
			area.proname AS provName,
			area.cid AS cityId,
			area.citname AS cityName,
			area.id AS areaId,
			area.areaName AS areaName,
			p.goods_transfee AS transFee,
			g.cdate,
			g.udate,
			DATE_FORMAT(SYSDATE(), '%Y-%m-%d %H:%i:%S') AS indexDate
		FROM
			t_goods_shop g
		INNER JOIN v_goods_product p ON g.productid = p.id
		LEFT JOIN v_ywd_goods_catalog_full vc ON p.gc_id = vc.gcid
		LEFT JOIN t_area_full area ON p.delivery_area_id = area.id
		LEFT JOIN t_goods_tag t ON g.goodscode = t.goodscode">

		<field column="id" name="id"  />
		<field column="goodsId" name="goodsId"  />
		<field column="productId" name="productId"  />
		<field column="goodsName" name="goodsName"  />
		<field column="goodsLogo" name="goodsLogo"  />
		<field column="goodsPrice" name="goodsPrice"  />
		<field column="goodsSales" name="goodsSales"  />
		<field column="goodsInventory" name="goodsInventory"  />
		<field column="status" name="status"  />
		<field column="fCateId" name="fCateId"  />
		<field column="fCateName" name="fCateName"  />
		<field column="sCateId" name="sCateId"  />
		<field column="sCateName" name="sCateName"  />
		<field column="tCateId" name="tCateId"  />
		<field column="tCateName" name="tCateName"  />
		<field column="provId" name="provId"  />
		<field column="provName" name="provName"  />
		<field column="cityId" name="cityId"  />
		<field column="cityName" name="cityName"  />
		<field column="areaId" name="areaId"  />
		<field column="areaName" name="areaName"  />
		<field column="brandId" name="brandId"  />
		<field column="brandName" name="brandName"  />
		<field column="transFee" name="transFee"  />
		<field column="cdate" name="cdate"  />
		<field column="udate" name="udate"  />
		<field column="indexDate" name="indexDate"  />
	</entity> 
  </document>
</dataConfig>

delta-data-config.xml 

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
  -->

<dataConfig>
 <dataSource name="gic_read" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
  url="jdbc:mysql://172.30.2.28:18806/gic?useUnicode=true&characterEncoding=utf-8"
      user="xxx" password="xxxx" batchSize="-1" />

 <document>
	<entity name="platformgoods" dataSource="gic_read" pk="id" query="SELECT
			CONCAT('', g.goodscode) AS id,
			g.goodscode AS goodsId,
			p.id AS productId,
			p.goods_name AS goodsName,
			IFNULL(p.logourl, '') AS goodsLogo,
			p.weixin_shop_price AS goodsPrice,
			g.goods_salenum AS goodsSales,
			p.goods_inventory AS goodsInventory,
			g.`status` AS `status`,
			g.shopid AS shopId,
			'1' AS goodsType,
			vc.fid AS fCateId,
			vc.fcatalog AS fCateName,
			vc.sid AS sCateId,
			vc.scatalog AS sCateName,
			vc.tid AS tCateId,
			vc.tcatalog AS tCateName,
			p.goods_brand_id AS brandId,
			p.`name` AS brandName,
			area.pid AS provId,
			area.proname AS provName,
			area.cid AS cityId,
			area.citname AS cityName,
			area.id AS areaId,
			area.areaName AS areaName,
			p.goods_transfee AS transFee,
			g.cdate,
			g.udate,
			DATE_FORMAT(SYSDATE(), '%Y-%m-%d %H:%i:%S') AS indexDate
		FROM
			t_goods_shop g
		INNER JOIN v_goods_product p ON g.productid = p.id
		LEFT JOIN v_ywd_goods_catalog_full vc ON p.gc_id = vc.gcid
		LEFT JOIN t_area_full area ON p.delivery_area_id = area.id
		LEFT JOIN t_goods_tag t ON g.goodscode = t.goodscode"
		
		deltaImportQuery="SELECT
			CONCAT('', g.goodscode) AS id,
			g.goodscode AS goodsId,
			p.id AS productId,
			p.goods_name AS goodsName,
			IFNULL(p.logourl, '') AS goodsLogo,
			p.weixin_shop_price AS goodsPrice,
			g.goods_salenum AS goodsSales,
			p.goods_inventory AS goodsInventory,
			g.`status` AS `status`,
			g.shopid AS shopId,
			'1' AS goodsType,
			vc.fid AS fCateId,
			vc.fcatalog AS fCateName,
			vc.sid AS sCateId,
			vc.scatalog AS sCateName,
			vc.tid AS tCateId,
			vc.tcatalog AS tCateName,
			p.goods_brand_id AS brandId,
			p.`name` AS brandName,
			area.pid AS provId,
			area.proname AS provName,
			area.cid AS cityId,
			area.citname AS cityName,
			area.id AS areaId,
			area.areaName AS areaName,
			p.goods_transfee AS transFee,
			g.cdate,
			g.udate,
			DATE_FORMAT(SYSDATE(), '%Y-%m-%d %H:%i:%S') AS indexDate
		FROM
			t_goods_shop g
		INNER JOIN v_goods_product p ON g.productid = p.id
		LEFT JOIN v_ywd_goods_catalog_full vc ON p.gc_id = vc.gcid
		LEFT JOIN t_area_full area ON p.delivery_area_id = area.id
		LEFT JOIN t_goods_tag t ON g.goodscode = t.goodscode
		WHERE g.goodscode = '${dih.delta.id}'"

		deltaQuery="SELECT
			g.goodscode AS id
		FROM
			t_goods_shop g
		INNER JOIN t_product p ON g.productid = p.hele_id
		LEFT JOIN v_ywd_goods_catalog_full vc ON p.gc_id = vc.gcid
		LEFT JOIN t_area_full area ON p.delivery_area_id = area.id
		LEFT JOIN t_goods_tag t ON g.goodscode = t.goodscode
		WHERE
			g.udate >= DATE_ADD('${dih.last_index_time}', INTERVAL - 3 SECOND)
		OR p.udate >= DATE_ADD('${dih.last_index_time}', INTERVAL - 3 SECOND)
		OR vc.udate >= DATE_ADD('${dih.last_index_time}', INTERVAL - 3 SECOND)
		OR t.udate >= DATE_ADD('${dih.last_index_time}', INTERVAL - 3 SECOND)"
		transformer="RegexTransformer">

		<field column="id" name="id"  />
		<field column="goodsId" name="goodsId"  />
		<field column="productId" name="productId"  />
		<field column="goodsName" name="goodsName"  />
		<field column="goodsLogo" name="goodsLogo"  />
		<field column="goodsPrice" name="goodsPrice"  />
		<field column="goodsSales" name="goodsSales"  />
		<field column="goodsInventory" name="goodsInventory"  />
		<field column="status" name="status"  />
		<field column="fCateId" name="fCateId"  />
		<field column="fCateName" name="fCateName"  />
		<field column="sCateId" name="sCateId"  />
		<field column="sCateName" name="sCateName"  />
		<field column="tCateId" name="tCateId"  />
		<field column="tCateName" name="tCateName"  />
		<field column="provId" name="provId"  />
		<field column="provName" name="provName"  />
		<field column="cityId" name="cityId"  />
		<field column="cityName" name="cityName"  />
		<field column="areaId" name="areaId"  />
		<field column="areaName" name="areaName"  />
		<field column="brandId" name="brandId"  />
		<field column="brandName" name="brandName"  />
		<field column="transFee" name="transFee"  />
		<field column="cdate" name="cdate"  />
		<field column="udate" name="udate"  />
		<field column="indexDate" name="indexDate"  />
	</entity> 
 </document>
</dataConfig>


漏了还要加个schema.xml，可以复制managed-schema来修改，

fuck solr 6.3直接用managed-schema，不用schema.xml。鄙视


<fieldType name="uuid" class="solr.UUIDField" sortMissingLast="true" indexed="true"/>

配置UUID，并删除原先的id域声明（<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />）

<field name="id" type="uuid" indexed="true" stored="true" required="true" multiValued="false" />

改变 id 生成策略


 <fieldType name="text_ik" class="solr.TextField" positionIncrementGap="100">
	<analyzer type="index" >
		<charFilter class="solr.HTMLStripCharFilterFactory"/>
		<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" />
		<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
		<filter class="solr.LowerCaseFilterFactory"/>
	</analyzer>
	<analyzer type="query">
        <charFilter class="solr.HTMLStripCharFilterFactory"/>
        <tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" dicPath="my_dic.dic"  />
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <filter class="solr.LowerCaseFilterFactory" />
	</analyzer> 
 </fieldType>


 <field name="fieldA" type="text_ik" indexed="false" stored="true"/>

<field name="id" type="string" indexed="true" stored="true" required="true" />
 <field name="goodsId" type="text_ws" indexed="true" stored="true" required="true" />
 <field name="productId" type="text_ws" indexed="true" stored="true" />
 <field name="goodsName" type="text_ws" indexed="false" stored="true" />
 <field name="goodsNameCp" type="lowercase" indexed="true" stored="true" />
 <copyField source="goodsName" dest="goodsNameCp"/>
 <field name="goodsLogo" type="text_ws" indexed="false" stored="true" />
 <field name="goodsPrice" type="tdouble" indexed="true" stored="true" />
 <field name="goodsSales" type="tint" indexed="true" stored="true" />
 <field name="goodsInventory" type="tint" indexed="true" stored="true" />
 <field name="status" type="text_ws" indexed="true" stored="true" />
 <field name="goodsType" type="text_ws" indexed="true" stored="true" />

 <field name="shopId" type="text_ws" indexed="true" stored="true" />

 <field name="fCateId" type="text_ws" indexed="true" stored="true" />
 <field name="fCateName" type="text_ws" indexed="false" stored="true" />
 <field name="sCateId" type="text_ws" indexed="true" stored="true" />
 <field name="sCateName" type="text_ws" indexed="false" stored="true" />
 <field name="tCateId" type="text_ws" indexed="true" stored="true" />
 <field name="tCateName" type="text_ws" indexed="false" stored="true" />

 <field name="provId" type="text_ws" indexed="true" stored="true" />
 <field name="provName" type="text_ws" indexed="false" stored="true" />
 <field name="cityId" type="text_ws" indexed="true" stored="true" />
 <field name="cityName" type="text_ws" indexed="true" stored="true" />
 <field name="areaId" type="text_ws" indexed="true" stored="true" />
 <field name="areaName" type="text_ws" indexed="false" stored="true" />
 
 <field name="brandId" type="text_ws" indexed="true" stored="true" />
 <field name="brandName" type="text_ws" indexed="false" stored="true" />
 <field name="transFee" type="int" indexed="true" stored="true" />
 <field name="cdate" type="tdate" indexed="false" stored="true" />
 <field name="udate" type="tdate" indexed="true" stored="true" />

 <field name="indexDate" type="text_ws" indexed="true" stored="true" />


在tomcat加上mysql数据库连接驱动jar


修改tomcat启动变量，创建一个新的配置col1216_cloud_conf到zookeepr配置管理




重启tomcat，以后修改col1216_cloud_conf里面的配置文件不需要重启tomcat，直接在solr页面collection reload按钮，其它两台机也是这样就可以同步配置文件夹。



不选platformgoods,导入的数据只有id,_version两个字段


yehuang_0801
关注
专栏目录
solr笔记
shuangmu9768的博客
09-06 1万+
【1】安装Solr7.0.1 【2】配置中文分词器 【3】从数据库导入 【4】solrCloud搭建 【5】基于tomcat的安全验证 【6】Solr的检索运算符 【7】查询 【8】fl参数详解 【9】solr函数 【10】设置solr/home的三种方式 【1】安装Solr7.0.1（解压后可用内嵌jetty直接运行，或者部署在tomcat中。）windows、linux均适用。 1.部署前的准备，安装jdk1.8及其以上版本，安装tomcat8及其以上版本。 2.拷贝源目录下server\solr-w
spring-cloud
weixin_44594263的博客
01-26 522
文章目录随堂笔记笔记乱码问题课程安排(17到18天)Spring Cloudspring cloud是什么课程要求业务模块注册和发现eureka作业Springboot 和 Springcloud 版本服务提供者sp02商品，和 sp05 eureka 集群远程调用RestTemplateRibbon负载均衡Ribbon 重试Hystrix降级Hystrix超时Hystrix 熔断Hystrix 故障监控 - Hystrix DashboardActuator搭建 Hystrix DashboardFeig
solrcloud配置dataimport
nethackatschool的专栏
11-05 2204
solrcloud配置dataimport full-import delta-import 上一篇说了solrcloud的中文分词，这一篇就讲讲solrcloud的dataimport。 很多搜索应用都将其待索引内容存储在结构化数据存储中，如关系型数据库。 数据导入处理器(DIH) 提供了从一个数据存储中导入内容并索引它的机制。 除了关系型数据库，
solr管理配置，关于数据导入，集群复制及日志配置
NickWar的专栏
11-28 8526
1 日志设置 默认solr的日志级别为为info，在使用过程中会产生大量日志，特别是多核情况下，启动速度和响应速度也会被拖慢 日志级别的设置方法，一是在admin页中临时设置，略去不表 另外就是在java虚拟机启动的参数中设置， 对于weblogic，找到weblogic使用的jdk目录， 在jre/lib/logging.properties文件中加入一行 org.apache.so
Solr集群搭建，zookeeper集群搭建，Solr分片管理，Solr集群下的DataImport，分词配置。
涂作权的博客
05-09 5957
1   什么是SolrCloud　　SolrCloud(solr 云)是Solr提供的分布式搜索方案，当你需要大规模，容错，分布式索引和检索能力时使用 SolrCloud。当一个系统的索引数据量少的时候是不需要使用SolrCloud的，当索引量很大，搜索请求并发很高，这时需要使　　用SolrCloud来满足这些需求。　　SolrCloud是基于Solr和Zookeeper的分布式搜索方案，它的
Solr6.5 集群搭建、数据库数据导入solr、分词器配置
weixin_34366546的博客
05-12 116
2019独角兽企业重金招聘Python工程师标准>>> ...
mysql集群 solr_13.5.SolrCloud集群使用手册之数据导入
weixin_42303955的博客
01-28 104
1.使用curl命令方式SolrCloud时会根据路由规则路由到各个shard。删除所有数据curl http://192.168.137.171:8080/solr-cloud/myc_shard1_replica1/update?commit=true -H "Content-Type: text/xml" --data-binary "*:*"导入XML文档数据curl http://192...
solr学习笔记
02-18
Solr 提供了一种称为 Data Import Handler (DIH) 的功能，用于从外部数据源（如数据库或 CSV 文件）导入数据到 Solr 集合。 1. **上传 JAR 包**: ```shell curl -X POST -H 'Content-Type: application/octet-...
关于SpringCloud的所有笔记
最新发布
weixin_51140686的博客
03-22 1629
关于SpringCloud
全文检索 solr
weixin_35994859的博客
02-04 599
全文检索 solr 课程计划 电商搜索案例分析 Solr的安装及配置 Solr整合tomcat Solr后台管理功能介绍 配置中文分析器 使用Solr的后台管理索引库 添加文档 删除文档 修改文档 查询文档 使用SolrJ管理索引库 添加文档 修改文档 删除文档 查询文档 电商搜索案例实现 案例分析 案例实现 1 什么是solr？ ​ Solr 是 Apache 下的一个顶级开源项目，采用 Java 开发，它是基于Lucene的全文搜索服务器，Solr 提供了比Lu
dataimport
08-05
导入数据到SQL SERVER
R Data Import/Export
09-27
Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the R Development Core Team.
solr5.5定时增量apache-solr-dataimportscheduler
04-09
经测试apache-solr-dataimportscheduler1.0 版本在solr5.5上已经不能使用，这是本人自己根据情况修改后打的jar包亲测可用
创建solr7.3.0集群_测试集群_添加中文分词器_导入数据
weixin_34148508的博客
04-25 206
2019独角兽企业重金招聘Python工程师标准>>> ...
solr 7 mysql导入_Solr7.4集群导入MySQL的数据
weixin_30843121的博客
01-19 81
1、将/usr/local/solr/solr-7.2.1/dist/jar包 复制到/usr/local/solr/tomcat/apache-tomcat-8.5.34-8081/webapps/solr/WEB-INF/lib/2、将mysql的jar包复制到/usr/local/solr/tomcat/apache-tomcat-8.5.34-8081/webapps/solr/WEB-I...
Solr dataimport 功能
weixin_33860737的博客
01-30 159
2019独角兽企业重金招聘Python工程师标准>>> ...
Java Solr：（七）Dataimport
地球村
09-25 310
Dataimport1.修改配置文件2.新建 data-config.xml3.添加 jar4.操作 可以使用 Solr 自带的 Dataimport 功能把数据库中数据快速导入到 solr 中 必须保证 managed-schema 和数据库中表的列对应 1.修改配置文件 修改 solrconfig.xml，添加下面内容 <!-- 配置数据导入的处理器 --> <requestHandler name="/dataimport" class="org.apache.solr.ha
Hadoop-Hbase-Solr 分布式集群部署手册（含中文分词）
本文档主要介绍了如何在分布式环境中部署Hadoop、Hbase和Solr，并集成了中文分词功能。文档适用于具有一定Linux基础的运维人员，提供了详细的集群节点规划、系统配置、软件安装和集成步骤。 在搭建这样一个分布式...
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值