提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档
Springboot集成Elasticsearch
前言
前面一节谈到了分词器,这一节详细说明Springboot集成Es
一、Es集成步骤
Springboot集成Es,使用Api方法的形式操作Es的数据
1.添加pom依赖
代码如下: 引入的版本和父工程有关,各个项目不同,都是大同小异
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
2.application.yml配置文件
代码如下:根据es部署的服务器,自行配置
es:
esip: 192.168.2.20
--username:暂时没配置
--password: 暂时没配置
3.Es配置文件类
代码如下: 配置文件类-(类似XXL-Job中的配置文件类和redis中的RedisTemplate)
提示: 像集成其他中间件或者工具都是需要自己的配置文件 如: RabbitMq,Redis,XXL-JOB,Ftp,DataSource
上面的配置文件只配置了esip, 没有配置端口和用户名密码,项目启动不会报错,这里也不需要用户名密码,服务器上的es是无密码的,如果设置了用户名和密码,在配置文件中加上即可
至于没配置不报错可参考前面的文章:
Springboot中@Value注解
@Configuration
public class ESConfig {
@Value("${es.esip:}")
private String esIp;
@Value("${es.port:}")
private String port;
@Value("${es.username:}")
private String username;
@Value("${es.password:}")
private String password;
@Bean
public RestHighLevelClient restHighLevelClient() {
if (StringUtil.isEmpty(esIp)) {
esIp = "127.0.0.1";
}
int port = 9200;
if (StringUtil.isNotEmpty(this.port)) {
try {
port = Integer.parseInt(this.port);
} catch (NumberFormatException e) {
}
}
HzRestHighLevelClient restHighLevelClient;
if (StringUtil.isNotEmpty(this.username, this.password)) {
// 使用 CredentialsProvider 对象登陆
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials
(AuthScope.ANY, new UsernamePasswordCredentials(this.username, this.password));
HttpHost http = new HttpHost(esIp, port, "http");
restHighLevelClient =
new HzRestHighLevelClient(RestClient.builder(
http)
.setHttpClientConfigCallback(
httpAsyncClientBuilder ->
httpAsyncClientBuilder.disableAuthCaching()
.setDefaultCredentialsProvider(credentialsProvider)
));
restHighLevelClient.ip = esIp;
restHighLevelClient.port = port;
restHighLevelClient.username = this.username;
restHighLevelClient.password = this.password;
restHighLevelClient.needAuth = Boolean.TRUE;
} else {
restHighLevelClient =
new HzRestHighLevelClient(RestClient.builder(
new HttpHost(esIp, port, "http")));
}
return restHighLevelClient;
}
public static class HzRestHighLevelClient extends RestHighLevelClient {
@Getter
private String username;
@Getter
private String password;
@Getter
private String ip;
@Getter
private int port;
@Getter
private boolean needAuth;
public HzRestHighLevelClient(RestClientBuilder restClientBuilder) {
super(restClientBuilder);
}
protected HzRestHighLevelClient(RestClientBuilder restClientBuilder, List<NamedXContentRegistry.Entry> namedXContentEntries) {
super(restClientBuilder, namedXContentEntries);
}
protected HzRestHighLevelClient(RestClient restClient, CheckedConsumer<RestClient, IOException> doClose, List<NamedXContentRegistry.Entry> namedXContentEntries) {
super(restClient, doClose, namedXContentEntries);
}
}
}
4.操作es的实体类
代码如下:注意indexName为索引名称,为属性title制定了分词器
@Data
@AllArgsConstructor
@NoArgsConstructor
@ToString
@Document(indexName = "es_search", type="_doc")
public class EsSearchEneity implements Serializable {
@Id
private Long id;
@Field(name="data_id",type = FieldType.Text)
private String dataId;
@Field(name="title", searchAnalyzer = "ik_smart", analyzer = "ik_max_word",type = FieldType.Text)
private String title;
@Field(name="type",type = FieldType.Keyword)
private String type;
@Field(name="en_type",type = FieldType.Keyword)
private String enType;
@Field(name="cus_number",type = FieldType.Keyword)
private String cusNumber;
}
提示:定义了两个实体类,idx_person为上一篇创建的索引
@Data
@AllArgsConstructor
@NoArgsConstructor
@ToString
@Document(indexName = "idx_person", type="_doc")
public class EsIdxPersonEntity implements Serializable {
@Id
private String id;
/**
* 人员ID
*/
@Field(name="person_id",type = FieldType.Text)
private String personId;
/**
* 人员编号
*/
@Field(name="person_no",type = FieldType.Text)
private String personNo;
@Field(name="person_name", searchAnalyzer = "search_pinyin_analyzer", analyzer = "pinyin_analyzer",type = FieldType.Text)
private String personName;
@Field(name="id_card",type = FieldType.Text)
private String idCard;
/**
* 人员类型
*/
@Field(name="person_type",type = FieldType.Integer)
private Integer personType;
@Field(name="cus_number",type = FieldType.Text)
private String cusNumber;
}
5.Controller操作es
代码如下:针对idx_person索引
@Api(tags = "ES搜索人员信息", description = "/person")
@RequestMapping("/person")
@RestController
public class EsIdxPersonController {
@Autowired
private EsIdxPersonService service;
@ApiOperation("查询人员信息")
@PostMapping("/searchPerson")
public R<List<EsIdxPersonEntity>> searchPerson(@RequestBody EsSearchPersonDto dto) throws IOException {
List<EsIdxPersonEntity> res = service.searchPerson(dto);
return R.ok(res);
}
@ApiOperation("将人员信息添加进ES")
@GetMapping("/savePersonData")
public R savePersonData() throws IOException {
service.savePersonData();
return R.ok();
}
}
6.Service
代码如下:
public interface EsIdxPersonService {
List<EsIdxPersonEntity> searchPerson(EsSearchPersonDto dto) throws IOException;
void savePersonData();
}
7.ServiceImpl
代码如下:
@Service
public class EsIdxPersonServiceImpl implements EsIdxPersonService {
private final static String indexName = "idx_person";
@Autowired
EsIdxPersonDao idxPersonDao;
@Autowired
EsIdxPersonMapper idxPersonMapper;
@Autowired
RestHighLevelClient client;
/**
* 目前支持 汉字、拼音全拼、拼音缩写、汉字拼音混合搜索人员姓名。
* 如果后期需要增加其他的查询条件,在此方法构建即可。
* 如果有其他的字段有需求需要更改索引的 Mapping。详细可以参考一下 doc目录下的 ES人员信息索引维护.docx
* @param dto
* @return
* @throws IOException
*/
@Override
public List<EsIdxPersonEntity> searchPerson(EsSearchPersonDto dto) throws IOException {
// 查询条件
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
if (!StringUtil.isBlank(dto.getPersonName())) {
boolQueryBuilder.must(QueryBuilders.matchPhraseQuery("person_name", dto.getPersonName()));
}
if (!StringUtil.isBlank(dto.getCusNumber())) {
boolQueryBuilder.must(QueryBuilders.termQuery("cus_number",dto.getCusNumber()));
}
if (dto.getPersonType() != null) {
boolQueryBuilder.must(QueryBuilders.termQuery("person_type", dto.getPersonType()));
}
// 将用户输入的姓名中的汉字拆分成一个个的汉字
String[] chineseList = StringUtil.extractChinese(dto.getPersonName());
if (!CollectionUtils.isEmpty(chineseList)) {
for (String chinese : chineseList) {
boolQueryBuilder.filter(QueryBuilders.termQuery("person_name", chinese));
}
}
return EsUtil.search(indexName,dto.getPageNum(),dto.getPageSize(),client,boolQueryBuilder,EsIdxPersonEntity.class);
}
/**
* 将人员信息维护进 ES文档中。此方法需要手动执行。
*/
public void savePersonData(){
idxPersonDao.deleteAll();
// 查询人员数据
List<EsIdxPersonEntity> esIdxPersonEneities = idxPersonMapper.queryAllPerson();
idxPersonDao.saveAll(esIdxPersonEneities);
}
}
8.Dao
代码如下:注意 dao是操作es的,可以把es当做数据库,是es的mapper
@Repository
public interface EsIdxPersonDao extends ElasticsearchRepository<EsIdxPersonEntity, String> {
}
9.mapper
代码如下:操作数据库的,通过mapper查询出数据,再通过dao存入es中
@Mapper
public interface EsIdxPersonMapper {
List<EsIdxPersonEntity> queryAllPerson();
}
10.mapper.xml
代码如下:操作数据库的,通过mapper查询出数据,再通过dao存入es中
<select id="queryAllPerson" resultType="com.hz.spp.es.entity.EsIdxPersonEntity">
select police_id id,police_id person_id,police_no person_no,police_name person_name,id_card, 3 "person_type", cus_number from plc_police_base_dtls
</select>
11.EsUtil
代码如下:操作数据库的,通过mapper查询出数据,再通过dao存入es中
/**
* elasticSearch工具类
*/
public class EsUtil {
/**
* 查询公共方法,查询条件在调用方构建
*
* @param indexName 索引名称
* @param pageNum 页码
* @param pageSize 页大小
* @param client 客户端
* @param queryBuilder 查询条件
* @param aClass 期待返回包装类的Class,如果为 null则返回 Map
* @return
*/
public static <T> List<T> search(String indexName, int pageNum, int pageSize, RestHighLevelClient client, QueryBuilder queryBuilder, Class<T> aClass) {
//1、条件搜索 参数 索引
SearchRequest searchRequest = new SearchRequest(indexName);
//2、构建搜索条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.from((pageNum - 1) * pageSize);
searchSourceBuilder.size(pageSize);
//3、注入执行查询条件
searchSourceBuilder.query(queryBuilder);
//4、设置查询超时时间
searchSourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
//5、执行查询 返回结果
searchRequest.source(searchSourceBuilder);
SearchResponse response = null;
try {
response = client.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
// if (client instanceof ESConfig.HzRestHighLevelClient && ((ESConfig.HzRestHighLevelClient) client).isNeedAuth()) {
// //账号密码登陆
// }
e.printStackTrace();
return null;
}
return Arrays.stream(response.getHits().getHits()).map(o -> JSON.parseObject(o.getSourceAsString(), aClass)).collect(Collectors.toList());
}
/**
* 查询公共方法,查询条件在调用方构建
*
* @param indexName 索引名称
* @param pageNum 页码
* @param pageSize 页大小
* @param client 客户端
* @param queryBuilder 查询条件
* @return
*/
public static List<Map<String, Object>> search(String indexName, int pageNum, int pageSize, RestHighLevelClient client, QueryBuilder queryBuilder) {
//1、条件搜索 参数 索引
SearchRequest searchRequest = new SearchRequest(indexName);
//2、构建搜索条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.from((pageNum - 1) * pageSize);
searchSourceBuilder.size(pageSize);
//3、注入执行查询条件
searchSourceBuilder.query(queryBuilder);
//4、设置查询超时时间
searchSourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
//5、执行查询 返回结果
searchRequest.source(searchSourceBuilder);
SearchResponse response = null;
try {
response = client.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
return null;
}
return Arrays.stream(response.getHits().getHits()).map(o -> o.getSourceAsMap()).collect(Collectors.toList());
}
/**
* 创建索引 判断索引是否存在,如果存在则返回true,如果不存在,则创建
* 创建成功返回true
*
* @return
* @throws IOException
*/
public static boolean createIndex(RestHighLevelClient client, String index) throws IOException {
//1、获取查询索引(库) 的请求
GetIndexRequest request = new GetIndexRequest(index);
//2、判断该索引是否存在
boolean flag_exist = client.indices().exists(request, RequestOptions.DEFAULT);
if (flag_exist) {
//如果存在该索引,则返回true
return true;
}
//如果不存在,该索引,则创建该索引
//3、创建新建索引(库) 的请求
CreateIndexRequest createIndexRequest = new CreateIndexRequest(index);
//4、执行请求,获得响应
CreateIndexResponse createIndexResponse = client.indices().create(createIndexRequest, RequestOptions.DEFAULT);
return createIndexResponse.isAcknowledged();
}
/**
* 从京东网页获取数据,批量插入elasticSearch数据
*
* @param key key是搜索jd的关键词
* @param index 索引库
* @return
* @throws IOException
*/
public static boolean insertListEs(RestHighLevelClient client, String key, String index) throws IOException {
//1、创建大批量数据插入请求
BulkRequest request = new BulkRequest();
//2、设置超时时间
request.timeout("10s");
//3、从京东网页中抓取数据,封装为实体类集合
List<EsSearchEneity> contents = HtmlParseUtil.getList(key);
//4、判断是否存在该索引
if (createIndex(client, index)) {
// 没有就创建,有就执行使用
//5、此时存在该索引,往该索引插入数据
for (EsSearchEneity content : contents) {
request.add(
new IndexRequest(index)
// .id(String.valueOf(content.getId())) # 重复的id会覆盖之前的数据 因为不是从数据库中查询,没有唯一的主键id,所以暂时不指定id
.source(JSON.toJSONString(content), XContentType.JSON));
}
BulkResponse response = client.bulk(request, RequestOptions.DEFAULT);
return !response.hasFailures();
}
return false;
}
/**
* 根据key关键词 搜索es中 title标题为key的商品信息(根据自己需求修改字段 title)
*
* @param client 高级客户端es
* @param index 索引
* @param key 关键词
* @param pageNum 分页页码
* @param pageSize 分页 每页的条数
* @return
* @throws IOException
*/
public static List<Map<String, Object>> searchEs(RestHighLevelClient client, String index, String key, int pageNum, int pageSize) throws IOException {
List<Map<String, Object>> list = new ArrayList<>();
//1、条件搜索 参数 索引
SearchRequest searchRequest = new SearchRequest(index);
//2、构建搜索条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//3、分页
int begin = (pageNum - 1) * pageSize;
searchSourceBuilder.from(begin);
searchSourceBuilder.size(pageSize);
//4、查询条件 全文搜索
MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("title", key);
//5、注入执行查询条件
searchSourceBuilder.query(matchQueryBuilder);
//6、设置查询超时时间
searchSourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
//7、执行查询 返回结果
searchRequest.source(searchSourceBuilder);
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(response);
for (SearchHit hit : response.getHits().getHits()) {
//遍历查询结果
System.out.println(hit.getSourceAsMap());
Map<String, Object> map = hit.getSourceAsMap();
System.out.println(map);
list.add(map);
}
//返回结果
return list;
}
/**
* 同上面的功能基本一样,添加了高亮显示功能
*
* @param client
* @param index
* @param key
* @param pageNum
* @param pageSize
* @return
* @throws IOException
*/
public static List<Map<String, Object>> searchTitleHighlight(RestHighLevelClient client, String index, String key, int pageNum, int pageSize, String cusNumber) throws IOException {
List<Map<String, Object>> list = new ArrayList<>();
//1、条件搜索 参数 索引
SearchRequest searchRequest = new SearchRequest(index);
//2、构建搜索条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//3、分页
int begin = (pageNum - 1) * pageSize;
searchSourceBuilder.from(begin);
searchSourceBuilder.size(pageSize);
//4、查询条件 全文搜索
BoolQueryBuilder matchQueryBuilder = QueryBuilders.boolQuery();
matchQueryBuilder.must(QueryBuilders.matchQuery("type", key));
if (cusNumber != null) {
matchQueryBuilder.must(QueryBuilders.matchQuery("cus_number", cusNumber));
}
// matchQueryBuilder.should(QueryBuilders.matchQuery("type", key));
//高亮
HighlightBuilder highlightBuilder = new HighlightBuilder();
highlightBuilder.field("title");
//前缀 后缀
highlightBuilder.preTags("<p class='light' style='color:red'>");
highlightBuilder.postTags("</p>");
searchSourceBuilder.highlighter(highlightBuilder);
highlightBuilder.requireFieldMatch(false);//一个文档只显示一个高亮
//5、注入执行查询条件
searchSourceBuilder.query(matchQueryBuilder);
//6、设置查询超时时间
searchSourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
//7、执行查询 返回结果
searchRequest.source(searchSourceBuilder);
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
for (SearchHit hit : response.getHits().getHits()) {
//遍历查询结果
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
HighlightField title = highlightFields.get("title");
Map<String, Object> map = hit.getSourceAsMap();//原来的结果
//解析高亮字段 将之前没有高亮的字段替换为现在高亮的字段即可
if (title != null) {
Text[] fragments = title.fragments();
String newTitle = "";
for (Text fragment : fragments) {
newTitle += fragment;
}
hit.getSourceAsMap().put("title", newTitle);
}
list.add(map);
}
//返回结果
return list;
}
public static List<Map<String, Object>> searchDetailHighlight(RestHighLevelClient client, String index, String key, int pageNum, int pageSize, String cusNumber) throws IOException {
SearchRequest searchRequest = new SearchRequest(index);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
MultiMatchQueryBuilder multiMatchQuery = QueryBuilders
.multiMatchQuery(key, "title", "type")
//默认是OR
.operator(Operator.AND);
HighlightBuilder highlightBuilder = new HighlightBuilder();
HighlightBuilder.Field highlightTitle = new HighlightBuilder.Field("title");
highlightBuilder.field(highlightTitle);
HighlightBuilder.Field highlightFilecontent = new HighlightBuilder.Field("type");
highlightBuilder.field(highlightFilecontent);
highlightBuilder
.preTags("<span style=color:red>")
.postTags("</span>");
searchSourceBuilder.highlighter(highlightBuilder);
searchSourceBuilder.query(multiMatchQuery);
//分页
int begin = (pageNum - 1) * pageSize;
searchSourceBuilder.from(begin);
searchSourceBuilder.size(pageSize);
searchRequest.source(searchSourceBuilder);
ArrayList<Map<String, Object>> resultList = new ArrayList<>();
SearchResponse searchResponse = client
.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = searchResponse.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
String source = hit.getSourceAsString();
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
HighlightField hTitle = highlightFields.get("title");
if (hTitle != null) {
String hBrandText = "";
Text[] fragments = hTitle.fragments();
for (Text text : fragments) {
hBrandText += text;
}
sourceAsMap.put("title", hBrandText);
}
// HighlightField hFilecontent = highlightFields.get("detail");
// if (hFilecontent != null) {
// String hNametText = "";
// Text[] fragments = hFilecontent.fragments();
// for (Text text : fragments) {
// hNametText += text;
// }
// sourceAsMap.put("detail", hNametText);
// }
resultList.add(sourceAsMap);
}
return resultList;
}
/**
* 根据条件等值匹配删除
*
* @param indexName 索引
* @param map 条件字段映射
* @param client
*/
public static long deleteEsData(String indexName, Map<String, Object> map, RestHighLevelClient client) throws IOException {
/*自定义条件删除:
通过QueryBuilders中的termQuery(等值匹配)、rangeQuery(范围匹配)、wildcardQuery(模糊匹配)指定搜索条件
通过QueryBuilders中的boolQuery中的should、must来设置and、or逻辑
通过DeleteByQueryRequest来构建删除请求,setQuery来装载条件,indices来指定索引
通过deleteByQuery来发起删除请求(es也是先查询后删除)
*/
DeleteByQueryRequest request = new DeleteByQueryRequest();
request.indices(indexName);
// 版本冲突
request.setConflicts("proceed");
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
Set<Map.Entry<String, Object>> entries = map.entrySet();
for (Map.Entry<String, Object> entry : entries) {
String key = entry.getKey();
Object value = entry.getValue();
/*
坑:
未加 .keyword发现删除BulkByScrollResponse->deleted为0
定位原因发现是应该 termsQuery进行了分词导致的
在出现分词查询,key 添加keyword,只适用于es6,加上keyword就不会进行分词了
可以修改 es 或者logstash 分词规则。比较好方式修改 es 映射规则
*/
boolQueryBuilder.must(QueryBuilders.termQuery(key + ".keyword", value));
}
request.setQuery(boolQueryBuilder);
// 删除后刷新
request.setRefresh(true);
// 删除数量
long deleted = 0L;
try {
BulkByScrollResponse response = client.deleteByQuery(request, RequestOptions.DEFAULT);
deleted = response.getDeleted();
} catch (IOException e) {
e.printStackTrace();
}
return deleted;
}
/**
* 根据条件等值匹配更新
*
* @param indexName 索引
* @param conditionMap 条件字段映射
* @param params 修改字段参数映射
* @param client
*/
public static long updateEsData(String indexName, Map<String, Object> conditionMap, Map<String, Object> params, RestHighLevelClient client) throws IOException {
/*自定义条件删除:
通过QueryBuilders中的termQuery(等值匹配)、rangeQuery(范围匹配)、wildcardQuery(模糊匹配)指定搜索条件
通过QueryBuilders中的boolQuery中的should、must来设置and、or逻辑
通过DeleteByQueryRequest来构建删除请求,setQuery来装载条件,indices来指定索引
通过deleteByQuery来发起删除请求(es也是先查询后删除)
*/
UpdateByQueryRequest request = new UpdateByQueryRequest();
request.indices(indexName);
// 版本冲突
request.setConflicts("proceed");
// 条件构造
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
Set<Map.Entry<String, Object>> entries = conditionMap.entrySet();
for (Map.Entry<String, Object> entry : entries) {
String key = entry.getKey();
Object value = entry.getValue();
/*
坑:
未加 .keyword发现删除BulkByScrollResponse->deleted为0
定位原因发现是应该 termsQuery进行了分词导致的
在出现分词查询,key 添加keyword,只适用于es6,加上keyword就不会进行分词了
可以修改 es 或者logstash 分词规则。比较好方式修改 es 映射规则
*/
boolQueryBuilder.must(QueryBuilders.termQuery(key + ".keyword", value));
}
request.setQuery(boolQueryBuilder);
// 脚本构造
StringBuilder script = new StringBuilder();
Set<String> keys = params.keySet();
for (String key : keys) {
String appendValue = "";
Object value = params.get(key);
if (value instanceof Number) {
appendValue = value.toString();
} else if (value instanceof String) {
appendValue = "'" + value.toString() + "'";
} else if (value instanceof List) {
appendValue = JSON.toJSONString(value);
} else {
appendValue = value.toString();
}
script.append("ctx._source.").append(key).append("=").append(appendValue).append(";");
}
request.setScript(new Script(script.toString()));
// 更新后刷新
request.setRefresh(true);
// 更新数量
long updated = 0L;
try {
BulkByScrollResponse response = client.updateByQuery(request, RequestOptions.DEFAULT);
updated = response.getUpdated();
} catch (IOException e) {
e.printStackTrace();
}
return updated;
}
}
总结
将数据库的数据查询出来,存入es中,前端查询数据,后端提供es的查询接口,从es中查询数据返回给前端,可以通过定时任务,定时存入最新数据到es中,有任何问题可私信~~!