版本控制
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-M74Okngl-1680862475871)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230403110845071.png)]
安装elasticsearch
简介
elasticsearch
下载
安装包下载地址:https://www.elastic.co/cn/downloads/past-releases#elasticsearch
解压
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ctOsKh6r-1680862475872)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172230290.png)]
启动
方式一:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IrO2qWwp-1680862475872)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406171852255.png)]
方式二:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IezmkhfD-1680862475873)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172357949.png)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qiGttJ86-1680862475873)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172545328.png)]
检查是否安装成功
访问:http://localhost:9200/
出现下面信息表示成功
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-c9WSawg3-1680862475874)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172759807.png)]
安装kibana
备注
是一个es可视化工具,功能很多,跟es一样,下载解压,双击bin下面的kibana.bat文件或者打开控制台输入kibana启动
访问kibana页面:http://localhost:5601/app/kibana
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-j97KxUjO-1680862475874)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180538887.png)]
语言设置转中文
网页纯英文,可自选设置成中文
编辑config下面的配置文件,文件最下端设置语言
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cbXHj6lM-1680862475875)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180331511.png)]
#英文
#i18n.locale: "en"
#中文
i18n.locale: "zh-CN"
es语法
参考资料:https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
rest风格
PUT //_doc/<_id>
POST //_doc/
PUT //_create/<_id>
POST //_create/<_id>
索引
创建索引
PUT /my-index-000001
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
}
PUT /my-index-000001
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
PUT /test
{
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"field1": { "type": "text" }
}
}
}
GET /my-index-000001/_mapping
修改索引
// 查看是否存在
HEAD my-data-stream
post /my-index-000001
获取索引
GET /my-index-000001
删除索引
DELETE /my-index-000001
文档
PUT /<target>/_doc/<_id> POST /<target>/_doc/ PUT /<target>/_create/<_id> POST /<target>/_create/<_id>
创建文档
POST my-index-000001/_doc/
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}
修改文档
POST //_update/<_id>
PUT test/_doc/1
{
"counter" : 1,
"tags" : ["red"]
}
获取文档
GET /_doc/<_id>
GET my-index-000001/_doc/0
GET my-index-000001/_doc/0?_source=false
删除文档
DELETE //_doc/<_id>
单个删除
DELETE /my-index-000001/_doc/1?timeout=5m
查询删除
// Deletes documents that match the specified query.
POST /my-index-000001/_delete_by_query
{
"query": {
"match": {
"user.id": "elkbee"
}
}
}
批量删除
POST /decode/resolve/_bulk
{"index":{"_id":"1"}}
{"vin": "D1445266366" }
{"index":{"_id":"2"}}
{"vin": "C4526365588" }
搜索
参考资料:
https://nasuyun.com/docs/api-search-text/api-search-text-multi-match
https://blog.csdn.net/qq_41911898/article/details/110089644
GET /<target>/_search GET /_search POST /<target>/_search POST /_search
字段匹配 最简单的搜索,使用match_all来表示,例如搜索全部
GET /decode/resolve/_search
{
"query": { "match_all": {} }
}
分页搜索,from表示偏移量,从0开始,size表示每页显示的数量
GET /decode/resolve/_search
{
"query": { "match_all": {} },
"from": 0,
"size": 10
}
搜索并返回指定字段内容,使用_source表示,例如只返回id和createTime两个字段内容
GET /decode/resolve/_search
{
"query": { "match_all": {} },
"sort": { "createTime": { "order": "desc" } }
}
条件搜索,使用match表示匹配条件,例如搜索出id为20的文档:
GET /decode/resolve/_search
{
"query": {
"match": {
"id": 20
}
}
}
短语匹配搜索,使用match_phrase表示,例如搜索address字段中同时包含mill和lane的文档:
GET /decode/resolve/_search
{
"query": {
"match_phrase": {
"address": "mill lane"
}
}
}
组合搜索,使用bool来进行组合,must表示同时满足,例如搜索address字段中同时包含mill和lane的文档
GET /decode/resolve/_search
{
"query": {
"bool": {
"must": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
9、组合搜索,should表示满足其中任意一个,搜索address字段中包含mill或者lane的文档
GET /decode/resolve/_search
{
"query": {
"bool": {
"should": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
10、组合搜索,must_not表示同时不满足,例如搜索address字段中不包含mill且不包含lane的文档
GET /decode/resolve/_search
{
"query": {
"bool": {
"must_not": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
11、组合搜索,组合must和must_not,例如搜索age字段等于40且state字段不包含ID的文档
GET /decode/resolve/_search
{
"query": {
"bool": {
"must": [
{ "match": { "age": "40" } }
],
"must_not": [
{ "match": { "state": "ID" } }
]
}
}
}
12、搜索过滤,使用filter来表示,例如过滤出balance字段在20000~30000的文档
GET /decode/resolve/_search
{
"query": {
"bool": {
"must": { "match_all": {} },
"filter": {
"range": {
"balance": {
"gte": 20000,
"lte": 30000
}
}
}
}
}
}
13、对搜索结果进行聚合,使用aggs来表示,类似于MySql中的group by,例如对vin字段进行聚合,统计出相同state的文档数量
GET /decode/resolve/_search
{
"size": 0,
"aggs": {
"group_by_vin": {
"terms": {
"field": "vin.keyword"
}
}
}
}
14、嵌套聚合,例如对state字段进行聚合,统计出相同vin的文档数量,再统计出balance的平均值
GET /decode/resolve/_search
{
"size": 0,
"aggs": {
"group_by_vin": {
"terms": {
"field": "vin.keyword"
},
"aggs": {
"average_balance": {
"avg": {
"field": "balance"
}
}
}
}
}
}
15、对聚合搜索的结果进行排序,例如按balance的平均值降序排列
GET /decode/resolve/_search
{
"size": 0,
"aggs": {
"group_by_vin": {
"terms": {
"field": "vin.keyword",
"order": {
"average_balance": "desc"
}
},
"aggs": {
"average_balance": {
"avg": {
"field": "balance"
}
}
}
}
}
}
16、按字段值的范围进行分段聚合,例如分段范围为age字段的[20,30] [30,40] [40,50],之后按gender统计文档个数和balance的平均值
GET /decode/resolve/_search
{
"size": 0,
"aggs": {
"group_by_age": {
"range": {
"field": "age",
"ranges": [
{
"from": 20,
"to": 30
},
{
"from": 30,
"to": 40
},
{
"from": 40,
"to": 50
}
]
},
"aggs": {
"group_by_gender": {
"terms": {
"field": "gender.keyword"
},
"aggs": {
"average_balance": {
"avg": {
"field": "balance"
}
}
}
}
}
}
}
}
分词
elasticsearch-head-master
简介
elasticsearch-head是一个前端项目,需要node环境,功能是作为一个es可视化界面。个人感觉比kibana更实用,可以看到索引,文档和进行增删改查,效果图如下
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-g99xoE3E-1680862475876)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180844170.png)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JTc31GxK-1680862475876)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406181107042.png)]
安装启动
下载依赖
npm i
启动
npm run start
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OZi36gKf-1680862475877)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406181516568.png)]
跨域
因为elasticsearch-head-master 是前端项目,访问elasticsearch 存在跨域问题,通过修改elasticsearch 配置解决
打开elasticsearch -> config -> elasticsearch.yml
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NAYUvg6L-1680862475878)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406182226585.png)]
拉到配置文件的最低端
#跨域问题
#设置可跨域
http.cors.enabled: true
#设置所有可访问
http.cors.allow-origin: "*"
springboot整合es
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SqJyEarX-1680862475879)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407140422589.png)]
环境
springboot == 2.4.4
elasticsearch==7.9.3
canal==1.1.5
导入依赖
<!-- elasticsearch -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<!-- spring -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
目录结构
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pMxDkG9r-1680862475879)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407161338552.png)]
添加配置
打开application.yml
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-6m8Fh51h-1680862475880)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406184751183.png)]
elasticsearch:
cluster:
name: http
host: 127.0.0.1
port: 9200
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1MI5fuY8-1680862475880)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406184815677.png)]
import org.apache.http.HttpHost;
import org.elasticsearch.client.ElasticsearchClient;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
/**
* @BelongsProject: es01
* @BelongsPackage: com.dhl.es01.client
* @Author: HuiLong.Ding
* @Description: TODO
* @Date: 2023/03/29 18:46
*/
@Configuration
public class ElasticSearchConfig {
private static final Logger logger = LoggerFactory.getLogger(ElasticsearchClient.class);
@Value("${elasticsearch.cluster.name}")
private String clusterName;
@Value("${elasticsearch.host}")
private String host;
@Value("${elasticsearch.port}")
private int port;
@Bean
public RestHighLevelClient restHighLevelClient(){
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost(host, port, clusterName)
)
);
logger.info("elasticsearch 连接成功");
return client;
}
}
java的es操作api
找官方文档 (7.6.x)
https://www.elastic.co/guide/index.html
索引
增删改查
import org.elasticsearch.client.RestHighLevelClient;
private RestHighLevelClient restHighLevelClient;
//-------------es rhlc 客户端
/**
* 创建索引
*
* @author HuiLong.Ding
* @date 2023/4/7 14:24
*/
@GetMapping("/index/put")
CreateIndexResponse putIndex() throws IOException {
// 1.创建索引请求
CreateIndexRequest request = new CreateIndexRequest("yd_pop_order");
// 2.执行创建请求,请求后获得响应createIndexResponse
CreateIndexResponse createIndexResponse = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
System.out.println("createIndexResponse = " + createIndexResponse);
return createIndexResponse;
}
// 测试获取索引,只能判断其是否存在
@GetMapping("/index/exist")
boolean existIndex() throws IOException {
// 1.创建索引请求
GetIndexRequest request = new GetIndexRequest("yd_pop_order");
// 2.获取索引,是否存在
return restHighLevelClient.indices().exists(request, RequestOptions.DEFAULT);
}
// 测试删除索引
@GetMapping("/index/del")
AcknowledgedResponse testDeleteIndex() throws IOException {
// 1.创建删除索引请求
DeleteIndexRequest request = new DeleteIndexRequest("yd_pop_order");
// 2.获取删除状态
AcknowledgedResponse delete = restHighLevelClient.indices().delete(request, RequestOptions.DEFAULT);
System.out.println(delete.isAcknowledged());
return delete;
}
// 测试添加文档
@GetMapping("/index/add")
IndexResponse testAddDocument() throws IOException {
// 1.创建对象
YdPopOrder ydPopOrder = new YdPopOrder();
ydPopOrder.setId(2609731592282L);
ydPopOrder.setState("6346453454");
ydPopOrder.setCreated(new Date());
ydPopOrder.setResponseJson("我是数据");
// 2.创建请求
IndexRequest request = new IndexRequest("yd_pop_order");
// 3.规则 put /yd_pop_order/_doc/1
request.id("1");
request.timeout(TimeValue.timeValueSeconds(1));// 或者 request.timeout("1s");
// 4.把数据放入请求 json
IndexRequest source = request.source(JSON.toJSONString(ydPopOrder), XContentType.JSON);
// 5.客户端发送请求,获取响应结果
IndexResponse indexResponse = restHighLevelClient.index(request, RequestOptions.DEFAULT);
System.out.println(indexResponse.toString());// 结果的json
System.out.println(indexResponse.status());// 对应我们命令返回的状态 created
return indexResponse;
}
// 获取文档的信息
@GetMapping("/index/get")
void testGetDocument() throws IOException {
GetRequest getRequest = new GetRequest("yd_pop_order", "1");
GetResponse getResponse = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
System.out.println(getResponse.getSourceAsString());// 打印文档的内容
System.out.println(getResponse);// 返回的全部内容和密令一样
}
文档
原生api参考资料:https://www.kuangstudy.com/bbs/1552226602677882882#header15
添加文档
@GetMapping("/put")
public boolean put() throws Exception {
YdPopOrder ydPopOrder = new YdPopOrder();
ydPopOrder.setId(2609731522282L);
ydPopOrder.setState("6346453454");
ydPopOrder.setCreated(new Date());
ydPopOrder.setResponseJson("数据");
ydPopOrderRepository.save(ydPopOrder);
return true;
}
批量添加
@GetMapping("/putall")
public boolean putall() throws Exception {
List<YdPopOrder> list = new ArrayList<>();
Date date = new Date();
for (int i = 0; i < 1000000; i++) {
YdPopOrder ydPopOrder = new YdPopOrder();
ydPopOrder.setId(2609731692282L + i);
ydPopOrder.setState("6346453454" + i);
ydPopOrder.setCreated(date);
ydPopOrder.setResponseJson("数据");
list.add(ydPopOrder);
}
ydPopOrderRepository.saveAll(list);
return true;
}
删除文档
@GetMapping("/del")
public boolean del(Long value) throws Exception {
ydPopOrderRepository.deleteById(value);
return true;
}
@GetMapping("/delall")
public boolean delall() throws Exception {
ydPopOrderRepository.deleteAll();
return true;
}
获取文档
@GetMapping("/get")
public Optional<YdPopOrder> get(Long value) throws Exception {
return ydPopOrderRepository.findById(value);
}
自定义
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-C9YwhFZy-1680862475881)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407141227855.png)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gKrSDz68-1680862475881)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407141300823.png)]
高亮
安装canal
下载无需安装,解压即可用,打开canal -> bin 替换startup.bat,然后双击startup.bat启动
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-furB6jHk-1680862475882)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406183554658.png)]
springboot整合canal
pom.xml导入依赖
<!-- canal -->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.5</version>
</dependency>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.protocol</artifactId>
<version>1.1.5</version>
</dependency>
application.yml
# canal
canal:
host: 127.0.0.1
port: 11111
destination: example
username: canal
password: canal
CanalClient
@Component
public class CanalClient implements DisposableBean {
private static final Logger logger = LoggerFactory.getLogger(CanalClient.class);
private CanalConnector canalConnector;
@Value("${canal.host}")
private String canalHost;
@Value("${canal.port}")
private String canalPort;
@Value("${canal.destination}")
private String canalDestination;
@Value("${canal.username}")
private String canalUsername;
@Value("${canal.password}")
private String canalPassword;
@Bean
public CanalConnector getCanalConnector() {
canalConnector = CanalConnectors.newClusterConnector(Lists.newArrayList(new InetSocketAddress(canalHost, Integer.valueOf(canalPort))), canalDestination, canalUsername, canalPassword);
canalConnector.connect();
// 指定filter,格式 {database}.{table},这里不做过滤,过滤操作留给用户
// .*\..* 可访问所有数据库的所有表 test\\..* 代表可访问名字为test的数据库下面的所有表
canalConnector.subscribe(".*\\..*");
// 回滚寻找上次中断的位置
canalConnector.rollback();
logger.info("canal客户端启动成功");
return canalConnector;
}
@Override
public void destroy() throws Exception {
if (canalConnector != null) {
canalConnector.disconnect();
}
}
}
注意 canalConnector.subscribe(“.\…”); 是对需要同步的数据库的过滤,可以通过正则匹配
参考资料:https://blog.csdn.net/Alice_qixin/article/details/108121895
全库全表 | connector.subscribe(“.\…”) |
---|---|
指定库全表 | connector.subscribe(“test\…*”) |
单表 | connector.subscribe(“test.user”) |
多规则组合使用 | connector.subscribe(“test\…*,test2.user1,test3.user2”) |
抛出问题
es
searchSimilar函数
时间格式问题: