注意点!!!
| 1.ES->默认使用分页查询,size为10,ES默认开启集群, 2.因为es是java开发的,所以使用的工具比如es,ik分词器,kibana的版本需要保持一致. 3.数据库有5000万条商品,ES有5000万条商品,哪个查询快? 回答:数据库快!ES是用来解决数据为PB级别的 问题的.操作数据库的语句叫SQL,操作ES的叫DSL. 4.ES的两个端口号: 9300->TCP端口号 9200->HTTP端口号(一般发送http请求或者通过浏览器访问). 5.ES默认只能查询10000调数据,当我们查询10001条数据时就直接报错,可以通过一下方法修改: |
我们如何理解ES的Index(索引)、Type(类型)、Field(域)、Document(文档)、Mapping(映射)?下图:
上图的"是否索引"是我们在查询数据的时候,我们通过条件查询数据的时候只有是索引的域才能当作查询条件.
上图熟记->String类型的域不能分词,StoreField(文档类型)的域只能存储.
分词器,索引域,文档域
| 1.存储数据的时候分词器要提取词语(进行分词),从而分词器影响索引域. 2.进行数据搜索的时候也会经过分词器,经过分词器提词然后到索引域找索引,最后根据索引域提供的索引找到数据. 3.分词器的工作流程(仅仅影响索引域): a.标准过滤->将无意义的字/词过滤掉,然后将标点符号去掉; b.大小写过滤->将所以的英文从大写转换为小写; c.停用词过滤->将文章中的停用词去掉(即索引域不会存储的词); d.词语提取. 4.查询数据的时候有两种情况: a.经过分词器->es会将查询条件进行分词; b.不经过分词器->es不会把查询条件进行分词. !!!!!!!!!!!!!!!!但是写入数据的时候一定会经过分词器,这也是es查询速度快的一大原因!!!!!!!!!!!!!!!!!!!!!!!! |
查询到的一条es的json数据
| _index | 文档存储的地方(数据库) |
| _type | 文档代表的对象的类(表) |
| _id | 文档的唯一标识 |
| mappings | 代表表结构 |
| _doc | 代表这个东西是一个表 |
| _score | 标识评分 |
| put person/_mapping{ "properties":{ "address":{ "type":"text" } } } | 在person库中添加表结构(针对于数据类型) |
| put person/_doc/1{ "address":"重庆" } | 往数据库里边添加数据,针对于数据本身, 1表示该_doc的id,每个_doc的id不能一样 |
| get person/-search | 查询person数据库的所有表 |
使用TCP方式创建索引库
/**
* 创建索引库
*
* @param args
*/
public static void main1(String[] args) throws Exception {
//客户端连接对象初始化,参数表示使用一个节点的es,不是用集群
TransportClient client = new PreBuiltTransportClient(Settings.EMPTY);
//设置需要连接的es的ip和端口号
client.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9300));
//创建索引->只创建索引库,没有mapping.记住!!但凡是带prepare的方法后边都加一个get()方法,如下
client.admin().indices().prepareCreate("java0509").get();
//关闭连接
client.close();
}
使用TCP方式创建索引库+mapping
//创建索引+映射
public static void main(String[] args) throws Exception {
//客户端连接对象初始化,参数表示使用一个节点的es,不是用集群
TransportClient client = new PreBuiltTransportClient(Settings.EMPTY);
//设置需要连接的es的ip和端口号
client.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9300));
//创建索引->只创建索引库,没有mapping.记住!!但凡是带prepare的方法后边都加一个get()方法,如下
client.admin().indices().prepareCreate("java0509_new").get();
//构建映射
XContentBuilder builder = XContentFactory.jsonBuilder();
builder
.startObject()
.startObject("article")
.startObject("properties")
.startObject("id")
.field("type","long")
.endObject()
.startObject("title")
.field("analyzer","ik_max_word")
.field("type","text")
.endObject()
.startObject("content")
.field("analyzer","ik_max_word")
.field("type","text")
.endObject()
.endObject()
.endObject()
.endObject();
//构建mapping请求对象
PutMappingRequest request = Requests.
putMappingRequest("java0509_new")
.type("article")
.source(builder);
//发送请求
client.admin().indices().putMapping(request);
//关闭连接
client.close();
}es整合(凡)
查询汇总
| 查询名称 | java代码 | 查询几次 | 是否分词 |
| 文档下标查询 | client.prepareGet() | 一 | |
| 查询所有 | QueryBuilders.matchAllQuery() | 一 | |
| 字符串查询 | QueryBuilders.queryStringQuery(条件).field(域) | 两 | 分 |
| 匹配查询 | QueryBuilders.matchQuery(域,Object条件) | 两 | 分 |
| 词条查询 | QueryBuilders.termQuery(域,Object条件) | 两 | 不分 |
| 模糊查询 | QueryBuilders.wildcardQuery(域,条件) | 两 | 不分 |
| 相似度查询 | QueryBuilders.fuzzyQuery(域,条件) | 两 | 不分 |
| 范围查询 | QueryBuilders.rangeQuery(域).gte().lte() | ||
| 组合查询 | QueryBuilders.booleanQuery() .must(QueryBuilders...) | ||
| 分页查询 | searchSourceBuilder对象.from().size() | ||
| 排序查询 | searchSourceBuilder对象.sort(域,规则) | ||
| 高亮查询 | |||
| 排序查询时 | SearchSourceBuilder对象.sort(字段,规则); 官方给了规则,不过我们不能写死,前端如果 想要升序,则会传递ASC字符串,我们拿到这个字符串之后, 使用SortOrder.valueOf(字符串)则可动态设置升降序. |
创建POJO和ES的dao层
说明:跟MP差不多,spring为我们创建了像BaseMapper这样的接口,里边整合了部分我们常用的api.
@Repository
public interface GoodsDao extends ElasticsearchRepository<Goods, Long> {
}
//启动类进行ES的dao层接口扫描
@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
@ComponentScan("com.atguigu.gmall")
//!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
@EnableReactiveElasticsearchRepositories(basePackages = "com.atguigu.gmall.list.dao")
@EnableFeignClients(basePackages = "com.atguigu.gmall.product.fegin")
public class ListApplication {
public static void main(String[] args) {
SpringApplication.run(ListApplication.class,args);
}
}
//相应的POJO
@Data
@Document(indexName = "goods" ,type = "info",shards = 3,replicas = 2)
public class Goods {
// 商品Id,就是_id
@Id
private Long id;
/**
* 如下,@Field注解里边的其中一个属性store,表示我们给ES的数据是否存储文档域,
* 属性值为false的话表示不存储,为true表示需要进行存储,那我们不存储的话作查询的时候数据从哪儿来?
* ES有一个隐藏的索引库,里边就是用来存数据的,所以我们insert进ES的数据都存储在这个隐藏库中,
* 做查询时就从这个隐藏库中取数据
*
*/
//Keyword:表示域的属性为字符串类型
//index:表示该字段在mapping中是否为索引
@Field(type = FieldType.Keyword, index = false ,store = false)
private String defaultImg;
//analyzer表示存储数据的时候使用什么分词器,searchAnalyzer表示查询数据的时候使用什么分词器
@Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_max_word")
private String title;
@Field(type = FieldType.Double)
private Double price;
//Date也是数字类型的域
@Field(type = FieldType.Date)
private Date createTime; // 新品
@Field(type = FieldType.Long)
private Long tmId;
@Field(type = FieldType.Keyword)
private String tmName;
@Field(type = FieldType.Keyword, index = false)
private String tmLogoUrl;
@Field(type = FieldType.Long)
private Long category1Id;
@Field(type = FieldType.Keyword)
private String category1Name;
@Field(type = FieldType.Long)
private Long category2Id;
@Field(type = FieldType.Keyword)
private String category2Name;
@Field(type = FieldType.Long)
private Long category3Id;
@Field(type = FieldType.Keyword)
private String category3Name;
@Field(type = FieldType.Long)
private Long hotScore = 0L;
// 平台属性集合对象
// Nested 支持嵌套查询
//Nested类型相当于一个对象,所以这个数据本身就含有自己的多个字段
//当我们使用该字段作查询条件进行查询的时候,可以使用 attrs.attrId = 8 的方式作查询条件
@Field(type = FieldType.Nested)
private List<SearchAttr> attrs;
}
//FieldType.Nested对应字段怎么进行查询条件构建?
String[] value = attrParam.getValue().split(":");
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must(QueryBuilders.termQuery("attrs.attrId", value[0]));
boolQueryBuilder.must(QueryBuilders.termQuery("attrs.attrValue", value[1]));
//!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
//构建平台属性查询条件
param.entrySet().stream().forEach(attrParam -> {
//获取前端传参的name属性
String attrIdAndName = attrParam.getKey();
//判断,如果name属性是以attr_开头,则把其设置为平台属性参数
if (attrIdAndName.startsWith("attr_")) {
String[] value = attrParam.getValue().split(":");
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must(QueryBuilders.termQuery("attrs.attrId", value[0]));
boolQueryBuilder.must(QueryBuilders.termQuery("attrs.attrValue", value[1]));
//使用nestedQuery(第一个参数:实体类中对应的字段名,第二个参数:8个基本查询,第三个参数:还为研究)
boolQuery.must(QueryBuilders.nestedQuery("attrs", boolQueryBuilder, ScoreMode.None));
}
});我们作查询的时候,查询一条数据用get,查询多条数据用search
聚合(聚合商品销售属性和销售属性值以及品牌)
// 平台属性相关对象
@Data
public class SearchResponseAttrVo implements Serializable {
// 平台属性Id
private Long attrId;//1
//当前属性值的集合
private List<String> attrValueList = new ArrayList<>();
//属性名称
private String attrName;//网络制式，分类
}
// 品牌数据
@Data
public class SearchResponseTmVo implements Serializable {
//当前属性值的所有值
private Long tmId;
//属性名称
private String tmName;//网络制式，分类
//图片名称
private String tmLogoUrl;//网络制式，分类
}
//设置查询条件
searchSourceBuilder.query(boolQuery);
//设置聚合条件,下边的aggTmId相当于tmId的别名 select tm_id as tmId from sku_info where name like "%手机%" group by tm_id;
searchSourceBuilder.aggregation(
AggregationBuilders.terms("aggTmId").field("tmId")
.subAggregation(AggregationBuilders.terms("aggTmName").field("tmName"))
.subAggregation(AggregationBuilders.terms("aggTmLogoUrl").field("tmLogoUrl")));
// 设置品台属性聚合条件
searchSourceBuilder.aggregation(
AggregationBuilders.nested("aggAttrs", "attrs")
.subAggregation(
AggregationBuilders.terms("aggAttrId").field("attrs.attrId")
.subAggregation(AggregationBuilders.terms("aggAttrName").field("attrs.attrName"))
.subAggregation(AggregationBuilders.terms("aggAttrValue").field("attrs.attrValue"))
.size(100)
)
);
//解析全部的聚合结果,此处的searchResponse为restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT)的返回值,即es返回给我们的查询结果;
Aggregations aggregations = searchResponse.getAggregations();
//解析品牌的聚合结果
List<SearchResponseTmVo> searchResponseTmVoList = getTmAggeResult(aggregations);
//解析品台属性的聚合结果
List<SearchResponseAttrVo> responseAttrVoList = getAttrAggResult(aggregations);
/**
* 解析品台属性的聚合
*
* @param aggregations
* @return
*/
private List<SearchResponseAttrVo> getAttrAggResult(Aggregations aggregations) {
//先解析nested
ParsedNested aggAttrs = aggregations.get("aggAttrs");
//获取子聚合查询结果:品台属性id的聚合结果
Aggregations nestedSubAggs = aggAttrs.getAggregations();
//获取品台属性id聚合结果的Buckets
ParsedLongTerms attrIdAggs = nestedSubAggs.get("aggAttrId");
List<SearchResponseAttrVo> searchResponseAttrVos = attrIdAggs.getBuckets().stream().map(attrIdAgg -> {
//返回结果初始化
SearchResponseAttrVo searchResponseAttrVo = new SearchResponseAttrVo();
//获取品台属性id
long attrId = ((Terms.Bucket) attrIdAgg).getKeyAsNumber().longValue();
//返回值设置品台属性id
searchResponseAttrVo.setAttrId(attrId);
//获取品台属性名
ParsedStringTerms attrNameAggs = ((Terms.Bucket) attrIdAgg).getAggregations().get("aggAttrName");
List<? extends Terms.Bucket> attrNameAggsBuckets = attrNameAggs.getBuckets();
if (attrNameAggsBuckets != null && !attrNameAggsBuckets.isEmpty()) {
//取出品台属性名
String attrName = attrNameAggsBuckets.get(0).getKeyAsString();
//返回值设置品台属性名
searchResponseAttrVo.setAttrName(attrName);
}
//获取品台属性值
ParsedStringTerms attrValueAggs = ((Terms.Bucket) attrIdAgg).getAggregations().get("aggAttrValue");
List<? extends Terms.Bucket> attrValueAggsBuckets = attrValueAggs.getBuckets();
if (attrValueAggsBuckets != null && !attrValueAggsBuckets.isEmpty()) {
List<String> attrValueList = attrValueAggsBuckets.stream().map(attrValueAgg -> {
return ((Terms.Bucket) attrValueAgg).getKeyAsString();
}).collect(Collectors.toList());
searchResponseAttrVo.setAttrValueList(attrValueList);
}
return searchResponseAttrVo;
}).collect(Collectors.toList());
return searchResponseAttrVos;
}
/**
* 解析品牌的聚合结果
*
* @param aggregations
* @return
*/
private List<SearchResponseTmVo> getTmAggeResult(Aggregations aggregations) {
//获取品牌的id
ParsedLongTerms aggTmId = aggregations.get("aggTmId");
//获取每条品牌id的聚合结果
List<SearchResponseTmVo> tmInfoList = aggTmId.getBuckets().stream().map(tmInfo -> {
SearchResponseTmVo searchResponseTmVo = new SearchResponseTmVo();
//获取品牌的id
//long tmId = ((Terms.Bucket) tmInfo).getKeyAsNumber().longValue(); ->凡
Long tmId = (Long) ((Terms.Bucket) tmInfo).getKey();
searchResponseTmVo.setTmId(tmId);
//品牌的名字集合
ParsedStringTerms aggTmName =
((Terms.Bucket) tmInfo).getAggregations().get("aggTmName");
//获取品牌的名字
if (aggTmName.getBuckets() != null && aggTmName.getBuckets().size() > 0) {
String tmName = aggTmName.getBuckets().get(0).getKeyAsString();
searchResponseTmVo.setTmName(tmName);
}
//品牌的Logo集合
ParsedStringTerms aggTmLogoUrl =
((Terms.Bucket) tmInfo).getAggregations().get("aggTmLogoUrl");
if (aggTmLogoUrl.getBuckets() != null && aggTmLogoUrl.getBuckets().size() > 0) {
String tmLogoUrl = aggTmLogoUrl.getBuckets().get(0).getKeyAsString();
searchResponseTmVo.setTmLogoUrl(tmLogoUrl);
}
return searchResponseTmVo;
}).collect(Collectors.toList());
return tmInfoList;
}es代码整合(马小马)->得使用RestHighLevelClient对象进行相关操作.
上边提到的RestHighLevelClient对象跟RabbitTemplate对象类似.
需写配置类
//host:127.0.0.1;port:9200
@Configuration
@ConfigurationProperties(prefix = "elasticsearch")
@Data
@ToString
public class ESConfig {
private String host;
private Integer port;
//创建ES的高级客户端,下方的"http"表示使用的协议
@Bean
public RestHighLevelClient client(){
return new RestHighLevelClient(RestClient.builder(new HttpHost(host,port,"http")));
}
}
以下所有方法中的client均为注入的RestHighLevelClient对象.
创建索引库,不声明数据结构
@Test
public void test01() throws Exception{
//获取操作索引库的对象
IndicesClient indices = client.indices();
CreateIndexRequest indexRequest = new CreateIndexRequest("person");
//发送请求
//第一个参数:表示创建的请求对象
//第二个参数:表示是否需要携带参数
CreateIndexResponse response = indices.create(indexRequest, RequestOptions.DEFAULT);
System.out.println(response.isAcknowledged());
}创建索引库,并声明数据结构
@Test
public void test02() throws Exception{
IndicesClient indices = client.indices();
CreateIndexRequest request = new CreateIndexRequest("person");
String mapping = "{\n" +
" \"properties\" : {\n" +
" \"address\" : {\n" +
" \"type\" : \"text\",\n" +
" \"analyzer\" : \"ik_max_word\"\n" +
" },\n" +
" \"age\" : {\n" +
" \"type\" : \"long\"\n" +
" },\n" +
" \"name\" : {\n" +
" \"type\" : \"keyword\"\n" +
" }\n" +
" }\n" +
" }";
request.mapping(mapping, XContentType.JSON);
CreateIndexResponse createIndexResponse = indices.create(request, RequestOptions.DEFAULT);
//以下打印为true则表示创建成功
System.out.println(createIndexResponse.isAcknowledged());
}查询索引库并获取数据结构
//查询索引库并获取表结构
@Test
public void test03() throws Exception{
IndicesClient indices = client.indices();
GetIndexRequest getIndexRequest = new GetIndexRequest("person");
GetIndexResponse response = indices.get(getIndexRequest, RequestOptions.DEFAULT);
//获取表结构
Map<String, MappingMetaData> mappings = response.getMappings();
for (String key : mappings.keySet()) {
System.out.println(key+":"+mappings.get(key).getSourceAsMap());
}
}删除索引库
//删除索引库
@Test
public void test04() throws Exception{
IndicesClient indices = client.indices();
DeleteIndexRequest abc = new DeleteIndexRequest("person");
AcknowledgedResponse delete = indices.delete(abc, RequestOptions.DEFAULT);
System.out.println(delete.isAcknowledged());
}判断索引库是否存在
//查询索引库是否存在
@Test
public void test05() throws Exception{
IndicesClient indices = client.indices();
GetIndexRequest abc = new GetIndexRequest("abc");
boolean exists = indices.exists(abc, RequestOptions.DEFAULT);
System.out.println(exists);
}往索引库里添加数据,添加的参数类型为map
@Test
public void test06() throws Exception{
Map param = new HashMap();
param.put("name","joker");
param.put("age",20);
param.put("address","深圳宝安");
IndexRequest personParam = new IndexRequest("person").id("1").source(param);
IndexResponse response = client.index(personParam, RequestOptions.DEFAULT);
System.out.println(response.getId());
}往索引库里边添加数据,添加的数据类型为JavaBean
@Test
public void test07() throws Exception{
Person person = new Person("2", "赵六", 31, "东北");
String data = JSON.toJSONString(person);
IndexRequest personParam = new IndexRequest("person").id("2").source(data,XContentType.JSON);
IndexResponse response = client.index(personParam, RequestOptions.DEFAULT);
System.out.println(response.getId());
}根据id进行查询操作
@Test
public void test08() throws Exception{
GetRequest person = new GetRequest("person").id("3");
GetResponse response = client.get(person, RequestOptions.DEFAULT);
System.out.println(response.getSourceAsString());
}根据id进行数据删除
@Test
public void test09() throws Exception{
DeleteRequest deleteRequest = new DeleteRequest("person").id("2");
DeleteResponse deleteResponse = client.delete(deleteRequest, RequestOptions.DEFAULT);
System.out.println(deleteResponse.getId()); //可以获取删除数据的id
System.out.println(deleteResponse.getIndex()); //可以获取数据数据的索引库
}批量操作,删1号数据,更改2号数据,添加3号数据
@Test
public void test10() throws Exception{
BulkRequest bulkRequest = new BulkRequest();
DeleteRequest deleteRequest = new DeleteRequest("person").id("1");
bulkRequest.add(deleteRequest);
Person person = new Person("3", "李四", 33, "南山");
String data = JSON.toJSONString(person);
IndexRequest indexRequest = new IndexRequest("person").id("3").source(data, XContentType.JSON);
bulkRequest.add(indexRequest);
//根据id进行更新,传入的更新数据为Map
Map param = new HashMap();
param.put("id","2");
param.put("name","田七");
param.put("age",33);
param.put("address","西北");
UpdateRequest updateRequest = new UpdateRequest("person", "2").doc(param);
bulkRequest.add(updateRequest);
BulkResponse bulkItemResponses = client.bulk(bulkRequest, RequestOptions.DEFAULT);
System.out.println(bulkItemResponses.status());
}分页查询
@Test
public void test12() throws Exception{
//下行代码表示全查询
MatchAllQueryBuilder matchAllQueryBuilder = QueryBuilders.matchAllQuery();
//设置查询条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(matchAllQueryBuilder);
searchSourceBuilder.from(0);
searchSourceBuilder.size(20);
//创建一个查询对象
SearchRequest searchRequest = new SearchRequest("goods");
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHit[] hits = searchResponse.getHits().getHits();
List<Goods> goodsList = new ArrayList<>();
for (SearchHit hit : hits) {
String sourceAsString = hit.getSourceAsString();
Goods goods = JSON.parseObject(sourceAsString, Goods.class);
goodsList.add(goods);
}
for (Goods goods : goodsList) {
System.out.println(goods);
}
}布尔查询->条件查询
//布尔查询,进行一些条件设置
//品牌:小米
//标题:手机
//价格:2000-3000之间
@Test
public void test13() throws Exception{
//创建查询对象
SearchRequest searchRequest = new SearchRequest("goods");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//多条件查询,使用布尔查询
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
//组装查询条件
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("brandName", "华为");
boolQueryBuilder.must(termQueryBuilder);
TermQueryBuilder termQueryBuilder1 = QueryBuilders.termQuery("title", "手机");
boolQueryBuilder.filter(termQueryBuilder1);
RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("price");
rangeQueryBuilder.gte(2000);
rangeQueryBuilder.lte(3000);
boolQueryBuilder.filter(rangeQueryBuilder);
searchSourceBuilder.query(boolQueryBuilder);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHit[] searchHits = searchResponse.getHits().getHits();
List<Goods> goodsList = new ArrayList<>();
for (SearchHit searchHit : searchHits) {
Goods goods = JSON.parseObject(searchHit.getSourceAsString(), Goods.class);
goodsList.add(goods);
}
for (Goods goods : goodsList) {
System.out.println(goods);
}
}
1391
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
