springDataElasticsearch +spring搭建
前言
在写这个博客的时候,我是很迷茫的,因为网上这类教程实在是多如牛毛,同时我自己也是看了很多的demo,但是我想要整理处理的是一个特别详细的教程。
遇到的坑
- java版本问题,
我这里使用的版本是:spring5.0.7.RELEASE+springDataElasticsearch3.0.7.RELEASE+jackson2.9.5 +elasticsearch-5.5.0
版本必须得对上,具体的就靠各位自己尝试
elasticsearch下载地址
- 死活链接不上es然后报错;
这个问题也是,当时我用的springDataElasticsearch3.0.7 版本对不上,修改各种配置文件都不行,最后找到一个版本对应图,下载对应版本就可以了
- linux启动报错
这个基本上没有难度,修改过一次文件打开数,就解决了具体的错误复制一下百度,就轻松解决 - 关于日期yyyy-MM-dd HH:mm:ss 排序报错,然后说没有字段没有设置fielddata=true,这简直就是个大坑,反正最后我在代码实体类类加注释、还设置es字段中的fielddata=true属性都没有解决。
最后解决办法是:吧字符串日期转换成Long类型,然后进行排序就可以了
Caused by: NotSerializableExceptionWrapper[: Fielddata is disabled on text fields by default. Set fielddata=true on [endDate] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.];
spring XML
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context" xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd http://www.springframework.org/schema/data/elasticsearch http://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch.xsd">
<!--开启包扫描 -->
<!-- <context:component-scan base-package="com.jingfuan.healthy.es.dao" /> -->
<!--配置集群信息 -->
<elasticsearch:transport-client id="esClient" cluster-name="elasticsearch" cluster-nodes="127.0.0.1:9300" client-transport-sniff="false"/>
<!--注入ESTemplate模板 -->
<bean id="elasticsearchTemplate" class="org.springframework.data.elasticsearch.core.ElasticsearchTemplate">
<constructor-arg name="client" ref="esClient" />
</bean>
<!--扫描Mapper -->
<elasticsearch:repositories base-package="com.jingfuan.healthy.es.dao" />
</beans>
dao类开发
这里的方法命名
可以参考springDataElasticsearch 官方文档 7.4:
springDataElasticsearch官方文档
package com.jingfuan.healthy.es.dao;
import java.util.Optional;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import com.jingfuan.healthy.visit.visitrecord.entity.BjtAppTVisitrecord;
public interface EsBjtAppTVisitrecordDao extends ElasticsearchRepository<BjtAppTVisitrecord, String> {
Optional<BjtAppTVisitrecord> findById(Integer id);
}
Service类
package com.jingfuan.healthy.es.service;
import java.util.List;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.sort.FieldSortBuilder;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.search.sort.SortOrder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.stereotype.Service;
import com.jingfuan.healthy.common.utils.StringUtils;
import com.jingfuan.healthy.es.dao.EsBjtAppTVisitrecordDao;
import com.jingfuan.healthy.visit.visitrecord.entity.BjtAppTVisitrecord;
@Service
public class EsBjtAppTVisitrecordService {
/**
* 日志对象
*/
protected Logger logger = LoggerFactory.getLogger(getClass());
@Autowired
private EsBjtAppTVisitrecordDao esBjtAppTVisitrecordDao;
/**
* 保存EsBjtAppTVisitrecord
*/
public void save(BjtAppTVisitrecord esBjtAppTVisitrecord) {
esBjtAppTVisitrecordDao.save(esBjtAppTVisitrecord);
}
/**
* 保存EsBjtAppTVisitrecord 集合
*/
public void saveAll(List<BjtAppTVisitrecord> esBjtAppTVisitrecord) {
esBjtAppTVisitrecordDao.saveAll(esBjtAppTVisitrecord);
}
/**
* 根据id 查找
*/
public boolean queryById(String id) {
return esBjtAppTVisitrecordDao.existsById(id);
}
/**
* 根据分页查询服务记录数据
*
* @param page
* 分页数据
* @return
*/
public com.jingfuan.healthy.common.persistence.Page<BjtAppTVisitrecord> EsPageBjtAppTVisitrecord(
com.jingfuan.healthy.common.persistence.Page<BjtAppTVisitrecord> page, BjtAppTVisitrecord bjtAppTVisitrecord) {
// 构建查询条件
// 中文查询
// match query搜索的时候,首先会解析查询字符串,进行分词,然后查询,而term
// query,输入的查询内容是什么,就会按照什么去查询,并不会解析查询内容,对它分词
// 如果有多个字符就使用matchQuery
// 如果只有一个字符如:'天' 使用termQuery
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
BoolQueryBuilder builder = QueryBuilders.boolQuery();
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getShopName())) {
// 模糊查询
builder.must(QueryBuilders.matchPhraseQuery("shopName", bjtAppTVisitrecord.getShopName()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getVisitorCard())) {
// 普通=查询
builder.must(QueryBuilders.termQuery("visitorCard.keyword", bjtAppTVisitrecord.getVisitorCard()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCard())) {
builder.must(QueryBuilders.termQuery("card.keyword", bjtAppTVisitrecord.getCard()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCounty())) {
builder.must(QueryBuilders.termQuery("county", bjtAppTVisitrecord.getCounty()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getProvince())) {
builder.must(QueryBuilders.termQuery("province", bjtAppTVisitrecord.getProvince()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCity())) {
builder.must(QueryBuilders.termQuery("city", bjtAppTVisitrecord.getCity()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getStreet())) {
builder.must(QueryBuilders.termQuery("street", bjtAppTVisitrecord.getStreet()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCommunity())) {
builder.must(QueryBuilders.termQuery("community", bjtAppTVisitrecord.getCommunity()));
}
if (bjtAppTVisitrecord.getHourlyPrice() != null) {
// 大于查询
builder.must(QueryBuilders.rangeQuery("hourlyPrice").gt(bjtAppTVisitrecord.getHourlyPrice()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getVisitorNumber())) {
builder.must(QueryBuilders.termQuery("visitorNumber", bjtAppTVisitrecord.getVisitorNumber()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getVisitContent())) {
builder.must(QueryBuilders.termQuery("visitContent", bjtAppTVisitrecord.getVisitContent()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceAuthenticity())) {
builder.must(QueryBuilders.termQuery("serviceAuthenticity", bjtAppTVisitrecord.getServiceAuthenticity()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceSatisfaction())) {
builder.must(QueryBuilders.termQuery("serviceSatisfaction", bjtAppTVisitrecord.getServiceSatisfaction()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceFirst())) {
builder.must(QueryBuilders.matchPhrasePrefixQuery("serviceType", bjtAppTVisitrecord.getServiceFirst() + "*"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceSecond())) {
builder.must(QueryBuilders.termQuery("serviceType", bjtAppTVisitrecord.getServiceSecond()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getIsEffective()) && "0".equals(bjtAppTVisitrecord.getIsEffective())) {
builder.must(QueryBuilders.termQuery("visitingStatus", "1"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getIsEffective()) && "1".equals(bjtAppTVisitrecord.getIsEffective())) {
// 不等于
builder.mustNot(QueryBuilders.termQuery("visitingStatus", "1"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getVisitingStatus())) {
builder.must(QueryBuilders.termQuery("visitingStatus", bjtAppTVisitrecord.getVisitingStatus()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getDomicile()) && "外埠".equals(bjtAppTVisitrecord.getDomicile())) {
// 不等于模糊查询
builder.mustNot(QueryBuilders.matchQuery("domicile", "北京"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getDomicile()) && "北京".equals(bjtAppTVisitrecord.getDomicile())) {
builder.must(QueryBuilders.matchQuery("domicile", bjtAppTVisitrecord.getDomicile()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCheck()) && "1".equals(bjtAppTVisitrecord.getCheck())) {
builder.must(QueryBuilders.termQuery("visitingStatus", "1"));
// is not null
builder.must(QueryBuilders.existsQuery("updateBys"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCheck()) && "2".equals(bjtAppTVisitrecord.getCheck())) {
builder.mustNot(QueryBuilders.termQuery("visitingStatus", "1"));
// is not null 查询
builder.must(QueryBuilders.existsQuery("updateBys"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceProviderType())) {
// in (value)查询
// String[] service_code =
// bjtAppTServceTypeDao.findByServiceProviderType(bjtAppTVisitrecord.getServiceProviderType());
// builder.must(QueryBuilders.termsQuery("serviceProviderType",
// service_code));
// 模糊查询
builder.must(QueryBuilders.wildcardQuery("serviceProviderType", "*" + bjtAppTVisitrecord.getServiceProviderType() + "*"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getServiceArea())) {
builder.must(QueryBuilders.termQuery("serviceArea", bjtAppTVisitrecord.getServiceArea()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getProviderCode())) {
builder.must(QueryBuilders.termQuery("providerCode", bjtAppTVisitrecord.getProviderCode()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getSuperiorcompany())) {
builder.must(QueryBuilders.termQuery("superiorcompany", bjtAppTVisitrecord.getSuperiorcompany()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCardOrBjt())) {
builder.must(QueryBuilders.termQuery("cardOrBjt", bjtAppTVisitrecord.getCardOrBjt()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getInnNum()) && "0".equals(bjtAppTVisitrecord.getInnNum())) {
// is null 查询
builder.mustNot(QueryBuilders.existsQuery("innNum"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getInnNum()) && "1".equals(bjtAppTVisitrecord.getInnNum())) {
// is not null 查询
builder.must(QueryBuilders.existsQuery("innNum"));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getCauseRecording())) {
builder.must(QueryBuilders.termQuery("causeRecording", bjtAppTVisitrecord.getCauseRecording()));
}
if (StringUtils.isNotBlank(bjtAppTVisitrecord.getPaytype())) {
builder.must(QueryBuilders.matchQuery("paytype", bjtAppTVisitrecord.getPaytype()));
}
if (bjtAppTVisitrecord.getStartDate() != null) {
// 大于等于查询
builder.must(QueryBuilders.rangeQuery("endDateLong").gte(bjtAppTVisitrecord.getStartDate().getTime()));
}
if (bjtAppTVisitrecord.getEndDate() != null) {
// 因为查询条件是yyyy-MM-dd 默认加上23:59:59==(1000 * 60 * 60 * 24 - 1000)
long endDate = bjtAppTVisitrecord.getEndDate().getTime() + (1000 * 60 * 60 * 24 - 1000);
// 小于等于查询
builder.must(QueryBuilders.rangeQuery("endDateLong").lte(endDate));
}
// 按照创建时间排序是依次降低
FieldSortBuilder sort = SortBuilders.fieldSort("endDate").order(SortOrder.DESC);
// 将排序设置到构建中
queryBuilder.withSort(sort);
// 构建查询
queryBuilder.withQuery(builder);
// 设置分页参数
queryBuilder.withPageable(PageRequest.of(page.getPageNo() - 1, page.getPageSize()));
Page<BjtAppTVisitrecord> espage = esBjtAppTVisitrecordDao.search(queryBuilder.build());
page.setList(espage.getContent());
page.setCount(espage.getTotalElements());
page.setPageNo(espage.getPageable().getPageNumber() + 1);
page.initialize();
logger.info("QueryDSL:\n{}", queryBuilder.build().getQuery().toString());
return page;
}
/**
* 通过id 删除实体
*/
public void delete(String id) {
esBjtAppTVisitrecordDao.deleteById(id);
}
public long count() {
return esBjtAppTVisitrecordDao.count();
}
}
实体类
/**
*
*/
package com.jingfuan.healthy.es.entity;
import org.springframework.data.elasticsearch.annotations.Document;
import com.jingfuan.healthy.visit.utils.VisitUtils;
@Document(indexName = "esbjtapptvisitrecord", type = "EsBjtAppTVisitrecord")
public class EsBjtAppTVisitrecord extends DataEntity<EsBjtAppTVisitrecord> {
get
set
字段
}
导入ES数据sql分页
目前我导入的700w数据使用oracle分页
select * from (
select * from aa
where 1=1 and rownum <= (${pageNo}-1)*${pageSize} + ${pageSize}
) table1 where table1.rowno > (${pageNo}-1)*${pageSize}
整理的查询条件
模糊查询
builder.must(QueryBuilders.matchPhraseQuery("shopName", bjtAppTVisitrecord.getShopName()));
汉字、字母普通=查询
builder.must(QueryBuilders.termQuery("visitorCard.keyword",bjtAppTVisitrecord.getVisitorCard()));
数字普通=查询
builder.must(QueryBuilders.termQuery("county",bjtAppTVisitrecord.getVisitorCard()));
大于查询
builder.must(QueryBuilders.rangeQuery("hourlyPrice").gt(bjtAppTVisitrecord.getHourlyPrice()));
不等于
builder.mustNot(QueryBuilders.termQuery("visitingStatus", "1"));
不等于模糊
builder.mustNot(QueryBuilders.matchQuery("domicile", "北京"));
is not null 查询
builder.must(QueryBuilders.existsQuery("updateBys"));
in
builder.must(QueryBuilders.termsQuery("serviceProviderType",service_code));
is null
builder.mustNot(QueryBuilders.existsQuery("innNum"));
is not null
builder.must(QueryBuilders.existsQuery("innNum"));
大于等于查询
builder.must(QueryBuilders.rangeQuery("endDateLong").gte(bjtAppTVisitrecord.getStartDate().getTime()));
小于等于查询
builder.must(QueryBuilders.rangeQuery("endDateLong").lte(endDate));