什么是 SolrCloud
SolrCloud(solr 云)是 Solr 提供的分布式搜索方案,当你需要大规模,容错,分布 式索引和检索能力时
使用 SolrCloud。当一个系统的索引数据量少的时候是不需要使用SolrCloud 的,当索引量很大,搜索请求
并发很高,这时需要使用 SolrCloud 来满足这些需求。
SolrCloud 是基于 Solr 和 Zookeeper 的分布式搜索方案,它的主要思想是使用
Zookeeper 作为集群的配置信息中心。
它有几个特色功能:
1)集中式的配置信息
2)自动容错
3)近实时搜索
4)查询时自动负载均衡
Solr 集群搭建设计
本次集群采用伪集群的方式进行安装,如果是真正的生产环境,建议搭建真实集群。
SolrCloud 结构图如下:
安装 Solr 集群环境
1需求:
1) 在 192.168.70.147 环境中安装 zookeeper 集群(已安装)
2) 创建 4 个 tomcat 实例,修改其端口为 8080-8083
3) 使用已安装好的单机版 solr 作为集群的节点使用
2创建 solrcloud 目录
mkdir solrcloud
3安装 Zookeeper 集群
步骤:
①使用文件上传工具把zookeeper资源包上传到linux上
②使用tar -zxf zookeeper-3.4.6.tar.gz 进行解压
③使用tar -zxf zookeeper-3.4.6.tar.gz 进行解压
④把解压好的文件拷贝到/usr/local目录下( cp zookeeper-3.4.6 /usr/local/zookeeper -r)
⑤拷贝conf目录下的zookeeper改名为zoo.cfg(zoo_sample.cfg为配置模板文件,zookeeper启动时默认加载conf/zoo.cfg配置文件)
⑥在zookeeper目录下创建data目录,用于存储缓存数据(mkdir data)
⑦在/usr/local下创建zookeepercluster目录
⑧复制3份zookeeper到zookeepercluster目录下
⑨在 data 目录中创建文件myid
方式一:touch myid创建文件,然后使用vi myid命令编辑文件,存储一个数字作为每个节点的唯一标识
方式二: echo [唯一标识] >> myid
⑩修改每一配置文件 zoo.cfg设置数据缓存路径,设置服务、投票、选举端口
clientPort=2181 #服务端口根据应用做对应修改,zk01-2181,zk02-2182,zk03-2183
server.1=192.168.116.131:2881:3881
server.2=192.168.116.131:2882:3882
server.3=192.168.116.131:2883:3883
启动脚本(关闭脚本类似)
步骤:
①使用vi/vim命令编辑一个后缀为.sh的文件(vi startAll.sh)
②配置启动命令
③使用chmod +x 文件名,给文件加上可执行的权限
启动 ZooKeeper 应用
./startAll.sh
访问 ZooKeeper 应用
bin/zkCli.sh -server 192.168.116.131:2181
4.安装 4 个 tomcat 实例并将 tomcat 与索引库拷贝到 solrCloud目录中
5修改端口号
vim tomact02/conf/server.xml
6.修改 solr 服务中指向 solr 索引库的路径
创建集群
1、上传索引库配置文件
把 solrhome 中的配置文件上传到 zookeeper 集群。使用 zookeeper 的客户端上传
cd /root/temp/solr-4.10.3/example/scripts/cloud-scripts/
上传文件
./zkcli.sh -zkhost 192.168.116.131:2181, 192.168.116.131:2182, 192.168.116.131:2183 -cmd upconfig -confdir
/usr/local/solrcloud/solrhome01/solr/collection1/conf -confname myconf
判断是否上传成功
步骤
cd /usr/local/solrcloud/zookeeper01/bin/
结果
2、修改 solrhome 下的 solr.xml 文件,指定当前实例运行的 ip 地址及端口号。
vim solrhome01/solr/solr.xml
3、修改每一台 solr 的 tomcat 的 bin 目录下 catalina.sh 文件中 加入 DzkHost 指定 zookeeper 服务器地址
cd /usr/local/solrcloud/tomact01/bin/
catalina.sh
JAVA_OPTS="-DzkHost=192.168.116.131:2181,192.168.116.131:2182,192.168.116.131:2183"
(可以使用 vim 的查找功能查找到 JAVA_OPTS 的定义的位置,然后添加) 注意不能含有空格
4、启动 tomcat
启动tomcat脚本
tomact01/bin/startup.sh
4、启动 tomcat
启动tomcat脚本
tomact01/bin/startup.sh
关闭tomcat脚本
tomact01/bin/shutdown.sh
http://192.168.116.131:8080/solr/#/~cloud
5、创建新的逻辑索引库并分片
创建一个新的 collection,并分两片,每片是一主一备。
使用以下命令创建:
http://192.168.116.131:8080/solr/admin/collections?action=CREATE&name=collection2&numShards=2&replicationFactor=2
6、删除原来的逻辑索引库
http://192.168.116.131:8080/solr/admin/collections?action=DELETE&name=collection1
集群中增删查
package com.sxt;
import java.io.IOException;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServer;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrServer;
import org.apache.solr.client.solrj.impl.HttpSolrServer;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
public class Test {
public static void main(String[] args) throws Exception {
//Test.solrCloudInsert();
Test.solrCloudDelete();
//Test.solrCloudSelect();
}
/**
* 向集群的索引库中添加文档
*
* @throws Exception
*/
public static void solrCloudInsert() throws Exception {
// zookeeper地址
String zkHost = "192.168.116.131:2181,192.168.116.131:2182,192.168.116.131:2183";
// 创建solrCloud对象
CloudSolrServer cloud = new CloudSolrServer(zkHost);
// 给定索引库
cloud.setDefaultCollection("collection2");
for (int i = 0; i < 20; i++) {
// 创建一个Solr的文档对象
SolrInputDocument docu = new SolrInputDocument();
// 向文档对象中添加需要插入的索引库的内容
docu.addField("id", "Old" + i);
docu.addField("item_title", "老好" + i);
docu.addField("item_price", 1000 + i);
// 将文档插入到solr的索引库中
cloud.add(docu);
}
// 事务提交
cloud.commit();
cloud.shutdown();
}
/**
* 删除集群中的文档
*
* @throws Exception
*/
public static void solrCloudDelete() throws Exception {
// zookeeper地址
String zkHost = "192.168.116.131:2181,192.168.116.131:2182,192.168.116.131:2183";
// 创建solrCloud对象
CloudSolrServer cloud = new CloudSolrServer(zkHost);
// 给定索引库
cloud.setDefaultCollection("collection2");
cloud.deleteByQuery("*:*");
// 事务提交
cloud.commit();
cloud.shutdown();
}
/**
* 查询集群中的文档
*
* @throws Exception
*/
public static void solrCloudSelect() throws Exception {
// zookeeper地址
String zkHost = "192.168.116.131:2181,192.168.116.131:2182,192.168.116.131:2183";
// 创建solrCloud对象
CloudSolrServer cloud = new CloudSolrServer(zkHost);
// 给定索引库
cloud.setDefaultCollection("collection2");
// 创建查询条件
SolrQuery sq = new SolrQuery();
sq.setQuery("老好了");
sq.set("df", "item_keywords");
// 设置分页
sq.setStart(0);
sq.setRows(10);
// 执行查询
// QueryResponse分装了查询结果集
QueryResponse query = cloud.query(sq);
SolrDocumentList results = query.getResults();
// results.getNumFound()数据总条数
System.out.println("总条数:" + results.getNumFound());
for (SolrDocument var : results) {
System.out.println(var.get("item_title"));
System.out.println(var.get("item_price"));
}
}
}
实战案例
案例需求:
1) 使用技术 springMVC+Spring+Mybatis+solrJ
2) 将 mysql 中的 tb_item 表中的部分业务数据导入到 solr 的索引库中
3) 提供一个搜索页面,在搜索页面中完成数据搜索
创建项目框架整合
1、创建maven项目'solrDemo2'类型war,父项目parent
2、pow.xml代码
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.sxt</groupId>
<artifactId>parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<groupId>com.sxt</groupId>
<artifactId>solrDemo2</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<dependencies>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
</dependency>
<!-- 单元测试 -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
<!-- 日志处理 -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<!-- Mybatis -->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
</dependency>
<!-- MySql -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<!-- 连接池 -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
</dependency>
<!-- Spring -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
</dependency>
<!-- JSP相关 -->
<dependency>
<groupId>jstl</groupId>
<artifactId>jstl</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jsp-api</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.xml</include>
</includes>
</resource>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.xml</include>
<include>**/*.properties</include>
</includes>
</resource>
</resources>
<!-- tomcat插件,由于子项目不一定每个都是web项目,所以该插件只是声明,并未开启 -->
<pluginManagement>
<plugins>
<!-- 配置Tomcat插件 -->
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<configuration>
<path>/</path>
<path>8080</path>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
3、项目框架整合
4、测试整合
Spring整合Slorj
创建 application-solrj.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:mvc="http://www.springframework.org/schema/mvc"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-4.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
<!-- 整合单机版slor
<bean id="httpSolrServer" class="org.apache.solr.client.solrj.impl.HttpSolrServer">
<constructor-arg name="baseURL">
<value>${SOLR_SERVICE_URL}</value>
</constructor-arg>
</bean> -->
<!-- 整合集群版 -->
<bean class="org.apache.solr.client.solrj.impl.CloudSolrServer">
<constructor-arg name="zkHost">
<value>${SOLR_CLOUD_SERVICE_URL}</value>
</constructor-arg>
<property name="defaultCollection">
<value>${DEFAULT_COLLECTION}</value>
</property>
</bean>
</beans>
编写junit测试TestSolrCloud
package com.sxt.solrDemo2;
import java.io.IOException;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrServer;
import org.apache.solr.common.SolrInputDocument;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class TestSolrColud {
/**
* 测试Spring整合SolrCloud
* @throws IOException
* @throws SolrServerException
*/
@Test
public void SolrCloudTest() throws Exception {
ApplicationContext ac = new ClassPathXmlApplicationContext("classpath:spring/application*");
CloudSolrServer cloud = ac.getBean(CloudSolrServer.class);
SolrInputDocument docu = new SolrInputDocument();
// 向文档对象中添加需要插入的索引库的内容
docu.addField("id", "Old");
docu.addField("item_title", "老好");
docu.addField("item_price", 1000);
// 将文档插入到solr的索引库中
cloud.add(docu);
// 事务提交
cloud.commit();
cloud.shutdown();
}
}
将 tb_item 表中的数据导入到 Solr 的索引库中
根据tb_item表创建实体类TbItem及其属性,并添加get和set方法
package com.sxt.pojo;
import java.util.Date;
public class TbItem {
private Long id;
private String title;
private String sell_Point;
private Long price;
private Integer num;
private String barcode;
private String image;
private Long cid;
private Byte status;
private Date created;
private Date updated;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title == null ? null : title.trim();
}
public String getSell_Point() {
return sell_Point;
}
public void setSell_Point(String sell_Point) {
this.sell_Point = sell_Point;
}
public Long getPrice() {
return price;
}
public void setPrice(Long price) {
this.price = price;
}
public Integer getNum() {
return num;
}
public void setNum(Integer num) {
this.num = num;
}
public String getBarcode() {
return barcode;
}
public void setBarcode(String barcode) {
this.barcode = barcode == null ? null : barcode.trim();
}
public String getImage() {
return image;
}
public void setImage(String image) {
this.image = image == null ? null : image.trim();
}
public Long getCid() {
return cid;
}
public void setCid(Long cid) {
this.cid = cid;
}
public Byte getStatus() {
return status;
}
public void setStatus(Byte status) {
this.status = status;
}
public Date getCreated() {
return created;
}
public void setCreated(Date created) {
this.created = created;
}
public Date getUpdated() {
return updated;
}
public void setUpdated(Date updated) {
this.updated = updated;
}
}
创建ItemMapper和ItemMapper.xml
ItemMapper
创建ImportItemService接口
创建实现类ImportItemServiceImpl
package com.sxt.servic.impl;
import java.util.ArrayList;
import java.util.List;
import org.apache.solr.client.solrj.impl.CloudSolrServer;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrInputDocument;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.sxt.mapper.ItemMapper;
import com.sxt.pojo.TbItem;
import com.sxt.servic.ImportItemService;
@Service
public class ImportItemServiceImpl implements ImportItemService{
@Autowired
private ItemMapper itemMapper;
@Autowired
private CloudSolrServer cloudSolrServer;
/**
* 导入数据
*/
@Override
public void importItem() {
try {
//查询数据库中数据
List<TbItem> list = this.itemMapper.findAll();
List<SolrInputDocument> result = new ArrayList<>();
//模型转换
for (TbItem tbItem : list) {
SolrInputDocument docu = new SolrInputDocument();
docu.setField("id", tbItem.getId()+"");
docu.setField("item_title", tbItem.getTitle());
docu.setField("item_sell_point", tbItem.getSell_Point());
docu.setField("item_price", tbItem.getPrice());
docu.setField("item_image", tbItem.getImage());
result.add(docu);
}
this.cloudSolrServer.add(result);
this.cloudSolrServer.commit();
}catch(Exception e){
e.printStackTrace();
}
}
}
创建导入数据 Controller
package com.sxt.web.controller;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import com.sxt.servic.ImportItemService;
@Controller
@RequestMapping("/import")
public class ImportDataController {
/**
* 处理导入数据的 Controller
*/
@Autowired
private ImportItemService importItemService;
@RequestMapping("/importData")
public String importData() {
this.importItemService.importItem();
return "ok";
}
}
删除原有数据
删除成功
测试添加数据
访问http://localhost:8080/import/importData
查询数据
实现搜索业务
创建搜索页面在index.jsp添加表单
<body>
<form action="/search/searchItem" method="post">
<input type="text" name="query"/>
<input type="submit" value="Search"/>
</form>
</body>
创建 SolrDao
实现类SolrDaoImpl
package com.sxt.dao.impl;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrServer;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;
import com.sxt.dao.SolrDao;
import com.sxt.pojo.PageResult;
import com.sxt.pojo.TbItem;
/**
* 查询solr
* @author 张瑞豪
*
*/
@Repository
public class SolrDaoImpl implements SolrDao {
@Autowired
private CloudSolrServer cloudSolrServer ;
@Override
public PageResult searchItem(SolrQuery solrQuery) throws Exception {
QueryResponse res = this.cloudSolrServer.query(solrQuery);
SolrDocumentList list = res.getResults();
//处理结果集
PageResult page = new PageResult();
//总条数
page.setTotalNum(list.getNumFound());
page.setTotalNum(list.getNumFound());
List<TbItem> items = new ArrayList<>();
//取高亮信息
Map<String, java.util.Map<String, List<String>>> hl = res.getHighlighting();
//模型转换
for(SolrDocument var :list){
TbItem item = new TbItem();
item.setId(Long.parseLong((String)var.get("id")));
item.setImage((String)var.get("item_image"));
item.setSell_Point((String)var.get("item_sell_point"));
item.setPrice((Long)var.get("item_price"));
List<String> h = hl.get(var.get("id")).get("item_title");
String title = "";
if(h != null && h.size() > 0){
title = h.get((0));
}else{
title= (String)var.get("item_title");
}
item.setTitle(title);
items.add(item);
}
page.setResult(items);
return page;
}
}
创建搜索 Service
实现类SearchItemServiceImpl
package com.sxt.servic.impl;
import org.apache.solr.client.solrj.SolrQuery;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.sxt.dao.SolrDao;
import com.sxt.pojo.PageResult;
import com.sxt.servic.SearchItemService;
@Service
public class SearchItemServiceImpl implements SearchItemService {
@Autowired
private SolrDao solrDao;
@Override
public PageResult searchItem(String query, Integer page, Integer rows) throws Exception{
//创建查询条件
SolrQuery solrQuery = new SolrQuery();
//添加查询条件
solrQuery.setQuery(query);
//默认域
solrQuery.set("df", "item_keywords");
//设置分页
solrQuery.setStart((page-1)*rows);
solrQuery.setRows(rows);
//设置高亮
solrQuery.setHighlight(true);
solrQuery.addHighlightField("item_title");
//设置高亮样式
solrQuery.setHighlightSimplePre("<em style='color:red;'>");
solrQuery.setHighlightSimplePost("</em>");
//调用SolrDao
PageResult result = this.solrDao.searchItem(solrQuery);
//补齐数据
//当前页
result.setPageIndex(page);
//总页数
Long total = result.getTotalNum()/rows;
if(result.getTotalNum() % rows > 0){
total++;
}
result.setTotalPage(total);
return result;
}
}
创建搜索 Controller
package com.sxt.web.controller;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import com.sxt.pojo.PageResult;
import com.sxt.servic.SearchItemService;
/**
* 处理商品搜索Controller
*/
@Controller
@RequestMapping("/seach")
public class SeachController {
@Autowired
private SearchItemService searchItemService;
/**
* 商品搜索
*/
@RequestMapping("/searchItem")
public String searchItem(String query,@RequestParam(value="page",defaultValue="1")Integer page, @RequestParam(value="rows",
defaultValue="20")Integer rows,Model model){
try{
PageResult result = this.searchItemService.searchItem(query, page, rows);
model.addAttribute("result", result);
}catch(Exception e){
e.printStackTrace();
}
return "showItem";
}
}
创建展示搜索结果页面JSP
<%@ page language="java" contentType="text/html; charset=UTF-8"
pageEncoding="UTF-8"%>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Insert title here</title>
</head>
<body>
<span>当前页:${result.pageIndex }</span>
<span>总页数:${result.totalPage }</span>
<span>总条数:${result.totalNum }</span>
<table align="center" border="1">
<c:forEach items="${result.result }" var="item">
<tr>
<td>${item.id }</td>
<td>${item.title }</td>
<td>${item.sell_Point }</td>
<td>${item.price }</td>
<td>${item.image }</td>
</tr>
</c:forEach>
</table>
</body>
</html>