【微服务全家桶】-实用篇-4-ES-上篇
Elasticsearch学习
1 初始Elasticsearch
1.1 Elatics
elasticsearch是一款非常强大的开源搜索引擎,可以帮助我们从海量数据中快速找到需要的内容。
elasticsearch结合kibana、Logstash、Beats,也就是elastic stack(ELK)。被广泛应用在日志数据分析、实时监控等领域。
数据可视化和数据抓取的技术是可替换的,只有es不可替换
elasticsearch具备下列优势:
-
支持分布式,可水平扩展
-
提供Restful接口,可被任何语言调用
1.2 正向索引和倒排索引
以mysql为例的正向索引
倒排索引
1.3 ES与Mysql对比
2 安装ES
详见CSDN中其他文章
3 索引库操作
3.1 mapping属性
mapping是对索引库中文档的约束,常见的mapping属性包括:
3.2 创建索引库
# 索引库映射
PUT /heima
{
"mappings": {
"properties": {
"info": {
"type": "text",
"analyzer": "ik_smart"
},
"email": {
"type": "keyword",
"index": false
},
"name": {
"properties": {
"firstName": {
"type": "keyword"
},
"lastName": {
"type": "keyword"
}
}
}
}
}
}
3.3 查看删除索引库
# 查看删除索引库
GET /heima
3.4 修改索引库
# 修改索引库-添加新字段
PUT /heima/_mapping
{
"properties": {
"age": {
"type":"integer"
}
}
}
4 文档操作
4.1 添加文档
# 插入文档
POST /heima/_doc/1
{
"info": "黑马程序员java讲师",
"email": "zy@itcast.cn",
"name": {
"firstName":"云",
"lastName" :"赵"
},
"age": 20
}
}
4.2 查询/删除文档
# 查询文档
GET /heima/_doc/1
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 3,
"_seq_no" : 2,
"_primary_term" : 5,
"found" : true,
"_source" : {
"info" : "黑马程序员java讲师",
"email" : "zy@itcast.cn",
"name" : {
"firstName" : "云",
"lastName" : "赵"
},
"age" : 20
}
}
#删除文档
DELETE /heima/_doc/1
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 4,
"result" : "deleted",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 3,
"_primary_term" : 5
}
再次查询
4.3 修改文档
4.3.1 全量修改
# 全量修改
PUT /heima/_doc/1
{
"info": "黑马程序员java讲师",
"email": "ZhaoYun@itcast.cn",
"name": {
"firstName": "云",
"lastName": "赵"
},
"age": 20
}
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 4,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 7,
"_primary_term" : 5
}
4.3.2 增量修改
# 增量修改
POST /heima/_update/1
{
"doc":{
"email": "ZYun@itcast.cn"
}
}
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 5,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 8,
"_primary_term" : 5
}
4.4 全局查询
只显示前十条
#查看酒店文档
GET /hotel/_search
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 201,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "hotel",
"_type" : "_doc",
"_id" : "36934",
"_score" : 1.0,
"_source" : {
"address" : "静安交通路40号",
"brand" : "7天酒店",
"business" : "四川北路商业区",
"city" : "上海",
"id" : 36934,
"location" : "31.251433, 121.47522",
"name" : "7天连锁酒店(上海宝山路地铁站店)",
"pic" : "https://m.tuniucdn.com/fb2/t1/G1/M00/3E/40/Cii9EVkyLrKIXo1vAAHgrxo_pUcAALcKQLD688AAeDH564_w200_h200_c1_t0.jpg",
"price" : 336,
"score" : 37,
"starName" : "二钻"
}
},
。。。
4.5 条件查询
#查看酒店文档
GET /hotel/_search?q=name:上海
hotel
索引中搜索包含在"name"字段中包含"上海"的文档
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 82,
"relation" : "eq"
},
"max_score" : 1.1531773,
"hits" : [
{
"_index" : "hotel",
"_type" : "_doc",
"_id" : "339777429",
"_score" : 1.1531773,
"_source" : {
"address" : "菊园新区嘉唐公路66号",
"brand" : "喜来登",
"business" : "嘉定新城",
"city" : "上海",
"id" : 339777429,
"location" : "31.394595, 121.245773",
"name" : "上海嘉定喜来登酒店",
"pic" : "https://m.tuniucdn.com/fb3/s1/2n9c/2v2fKuo5bzhunSBC1n1E42cLTkZV_w200_h200_c1_t0.jpg",
"price" : 1286,
"score" : 44,
"starName" : "五钻"
}
},
。。。
5 RestClient
ES官方提供了各种不同语言的客户端,用来操作ES。这些客户端的本质就是组装DSL语句,通过http请求发送给ES。
5.1 RestClient操作索引库
5.1.1步骤一 导入表和demo文件
hotel-demo中的application.yaml
server:
port: 8089
spring:
datasource:
url: jdbc:mysql://mysql:3306/heima?useSSL=false
username: root
password: 123sjbsjb
driver-class-name: com.mysql.jdbc.Driver
logging:
level:
cn.itcast: debug
pattern:
dateformat: MM-dd HH:mm:ss:SSS
mybatis-plus:
configuration:
map-underscore-to-camel-case: true
type-aliases-package: cn.itcast.hotel.pojo
5.1.2 步骤二 分析数据结构 创建mapping
#酒店的mapping
PUT /hotel
{
"mappings":{
"properties": {
"id": {
"type": "keyword",
"copy_to": "all"
},
"name":{
"type": "text",
"analyzer": "ik_max_word"
},
"address":{
"type": "keyword",
"index":false
},
"price":{
"type": "integer"
},
"score":{
"type": "integer"
},
"brand":{
"type": "keyword",
"copy_to": "all"
},
"city":{
"type": "keyword"
},
"starName":{
"type": "keyword"
},
"business":{
"type": "keyword",
"copy_to": "all"
},
"location":{
"type": "geo_point"
},
"pic":{
"type": "keyword",
"index":false
},
"all":{
"type": "text",
"analyzer": "ik_max_word"
}
}
}
}
5.1.3 初始化JavaRestClient
1.引入es的RestHighLevelCLient依赖:
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
</dependency>
2.因为SpringBoot默认的ES版本是7.6.2,所以我们需要覆盖默认的ES版本:
<properties>
<java.version>1.8</java.version>
<elasticsearch.version>7.12.1</elasticsearch.version>
</properties>
3.初始化RestHighLevelClient:
在test/java/cn.itcast/hotel下创建HotelIndexTest
使用**@BeforeEach和@AfterEach**可以在测试方法前创建对象
public class HotelIndexTest {
private RestHighLevelClient client;
@Test
public void testCreateIndex() {
System.out.println(client);
}
//提前初始化RestHighLevelClient
@BeforeEach
public void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.204.129:9200"))
);
}
//销毁
@
public void tearDown() {
try {
client.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
5.1.4 创建索引库
创建索引库代码
@Test
void testCreateHotelIndex() {
//1.创建Request对象
CreateIndexRequest request = new CreateIndexRequest("hotel");
//2.请求参数,MAPPING_TEMPLATE是静态常量字符串,内容是创建索引库的DSL语句
request.source(MAPPING_TEMPLATE, XContentType.JSON);
//3.发起请求
try {
client.indices().create(request, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
}
其中MAPPING_TEMPLATE是个静态常量,在cn.itcast.hotel下创建constants.HotelConstants,其中是对数据中的索引库创建的语句
public class HotelConstants {
public static final String MAPPING_TEMPLATE="{\n" +
" \"mappings\":{\n" +
" \"properties\": {\n" +
" \"id\": {\n" +
" \"type\": \"keyword\",\n" +
" \"copy_to\": \"all\"\n" +
" },\n" +
" \"name\":{\n" +
" \"type\": \"text\",\n" +
" \"analyzer\": \"ik_max_word\"\n" +
" },\n" +
" \"address\":{\n" +
" \"type\": \"keyword\",\n" +
" \"index\":false\n" +
" },\n" +
" \"price\":{\n" +
" \"type\": \"integer\"\n" +
" },\n" +
" \"score\":{\n" +
" \"type\": \"integer\"\n" +
" },\n" +
" \"brand\":{\n" +
" \"type\": \"keyword\",\n" +
" \"copy_to\": \"all\"\n" +
" },\n" +
" \"city\":{\n" +
" \"type\": \"keyword\"\n" +
" },\n" +
" \"starName\":{\n" +
" \"type\": \"keyword\"\n" +
" },\n" +
" \"business\":{\n" +
" \"type\": \"keyword\",\n" +
" \"copy_to\": \"all\"\n" +
" },\n" +
" \"location\":{\n" +
" \"type\": \"geo_point\"\n" +
" },\n" +
" \"pic\":{\n" +
" \"type\": \"keyword\",\n" +
" \"index\":false\n" +
" },\n" +
" \"all\":{\n" +
" \"type\": \"text\",\n" +
" \"analyzer\": \"ik_max_word\"\n" +
" }\n" +
" }\n" +
" }\n" +
"}";
}
查看索引库
#查看酒店索引库
GET /hotel
{
"hotel" : {
"aliases" : { },
"mappings" : {
"properties" : {
"address" : {
"type" : "keyword",
"index" : false
},
"all" : {
"type" : "text",
"analyzer" : "ik_max_word"
},
"brand" : {
"type" : "keyword",
"copy_to" : [
"all"
]
},
"business" : {
"type" : "keyword",
"copy_to" : [
"all"
]
},
"city" : {
"type" : "keyword"
},
"id" : {
"type" : "keyword",
"copy_to" : [
"all"
]
},
"location" : {
"type" : "geo_point"
},
"name" : {
"type" : "text",
"analyzer" : "ik_max_word"
},
"pic" : {
"type" : "keyword",
"index" : false
},
"price" : {
"type" : "integer"
},
"score" : {
"type" : "integer"
},
"starName" : {
"type" : "keyword"
}
}
},
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "hotel",
"creation_date" : "1708945726260",
"number_of_replicas" : "1",
"uuid" : "4L8tS8p4QqGUJrYqWZGozQ",
"version" : {
"created" : "7120199"
}
}
}
}
}
5.1.5 删除索引库、判断索引库是否存在
删除索引库
//删除索引库
@Test
void testDeleteHotelIndex() {
//1.创建Request对象
DeleteIndexRequest request = new DeleteIndexRequest("hotel");
//2.发起请求
try {
client.indices().delete(request, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
}
判断索引库是否存在
//判断索引库是否存在
@Test
void testExistHotelIndex() {
//1.创建Request对象
GetIndexRequest request = new GetIndexRequest("hotel");
//2.发起请求
try {
boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);
System.err.println(exists?"索引库存在":"索引库不存在");
} catch (IOException e) {
e.printStackTrace();
}
}
5.2 RestClient操作文档
5.2.1 初始化JavaRestClient
在测试类中创建HotelDocumentIndexTest
public class HotelIndexTest {
private RestHighLevelClient client;
@Test
public void testCreateIndex() {
System.out.println(client);
}
//提前初始化RestHighLevelClient
@BeforeEach
public void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.204.129:9200"))
);
}
//销毁
@
public void tearDown() {
try {
client.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
5.2.2 添加酒店数据到数据库
先查询酒店数据,然后给这条数据创建倒排索引,即可完成添加
采用mp的写法,注入IHotelService hotelService,之后用hotelService访问数据库
@SpringBootTest
public class HotelDocumentIndexTest {
@Autowired
private IHotelService hotelService;
private RestHighLevelClient client;
@Test
public void testCreateIndex() {
System.out.println(client);
}
//添加文档
@Test
void testAddIndexDocument() throws IOException {
//根据ID查询
Hotel hotel = hotelService.getById(61083L);
//转换为文档类型
HotelDoc hotelDoc = new HotelDoc(hotel);
//1.创建Request对象
IndexRequest request = new IndexRequest("hotel").id(hotel.getId().toString());
//2.准备JSON文档
request.source(JSON.toJSONString(hotelDoc), XContentType.JSON);
//3.发起请求
client.index(request, RequestOptions.DEFAULT);
}
//提前初始化RestHighLevelClient。。。
//销毁。。。
}
插入后查看
#查看酒店文档
GET /hotel/_doc/61083
{
"_index" : "hotel",
"_type" : "_doc",
"_id" : "61083",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"address" : "自由贸易试验区临港新片区南岛1号",
"brand" : "皇冠假日",
"business" : "滴水湖临港地区",
"city" : "上海",
"id" : 61083,
"location" : "30.890867, 121.937241",
"name" : "上海滴水湖皇冠假日酒店",
"pic" : "https://m.tuniucdn.com/fb3/s1/2n9c/312e971Rnj9qFyR3pPv4bTtpj1hX_w200_h200_c1_t0.jpg",
"price" : 971,
"score" : 44,
"starName" : "五钻"
}
}
5.2.3 根据id查询酒店数据
因为查询到的文档是json,所以需要反序列化成java对象
//查询文档
@Test
void testFindIndexDocument() throws IOException {
//1.创建Request对象
GetRequest request = new GetRequest("hotel","61083");
//2.发起请求
GetResponse response = client.get(request, RequestOptions.DEFAULT);
//3.发起请求
String json= response.getSourceAsString();
HotelDoc hotelDoc = JSON.parseObject(json, HotelDoc.class);
System.out.println(hotelDoc);
}
HotelDoc(id=61083, name=上海滴水湖皇冠假日酒店, address=自由贸易试验区临港新片区南岛1号, price=971, score=44, brand=皇冠假日, city=上海, starName=五钻, business=滴水湖临港地区, location=30.890867, 121.937241, pic=https://m.tuniucdn.com/fb3/s1/2n9c/312e971Rnj9qFyR3pPv4bTtpj1hX_w200_h200_c1_t0.jpg)
5.2.4 根据id修改酒店数据
//局部更新
@Test
void testUpdateIndexDocument() throws IOException {
//1.创建Request对象
UpdateRequest request = new UpdateRequest("hotel","61083");
//2.准备JSON文档
request.doc(
"price","952",
"starName","四钻"
);
//3.发起请求
client.update(request, RequestOptions.DEFAULT);
}
HotelDoc(id=61083, name=上海滴水湖皇冠假日酒店, address=自由贸易试验区临港新片区南岛1号, price=952, score=44, brand=皇冠假日, city=上海, starName=四钻, business=滴水湖临港地区, location=30.890867, 121.937241, pic=https://m.tuniucdn.com/fb3/s1/2n9c/312e971Rnj9qFyR3pPv4bTtpj1hX_w200_h200_c1_t0.jpg)
5.2.5 根据id删除酒店数据
//删除酒店数据
@Test
void testDeleteIndexDocument() throws IOException {
//1.创建Request对象
DeleteRequest request = new DeleteRequest("hotel","61083");
//2.发起请求
client.delete(request, RequestOptions.DEFAULT);
}
5.3 JavaRestClient批量导入
每次添加发送请求
//批量导入酒店数据
@Test
void testBatchInsertIndexDocument(){
//查询所有酒店数据
hotelService.list().forEach(hotel -> {
//转换为文档类型
HotelDoc hotelDoc = new HotelDoc(hotel);
//1.创建Request对象
IndexRequest request = new IndexRequest("hotel").id(hotel.getId().toString());
//2.准备JSON文档
request.source(JSON.toJSONString(hotelDoc), XContentType.JSON);
//3.发起请求
try {
client.index(request, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
});
}
利用bulk发起请求
//批量导入数据利用client的bulk方法
@Test
void testBatchInsertIndexDocument2() throws IOException {
//1.创建BulkRequest对象
BulkRequest request = new BulkRequest();
//2.准备数据,添加到BulkRequest对象中
//查询所有酒店数据
hotelService.list().forEach(hotel -> {
//转换为文档类型
HotelDoc hotelDoc = new HotelDoc(hotel);
//3.添加到BulkRequest对象中
request.add(new IndexRequest("hotel")
.id(hotel.getId().toString())
.source(JSON.toJSONString(hotelDoc), XContentType.JSON));
});
//4.发起请求
client.bulk(request, RequestOptions.DEFAULT);
}