题记
Elasticsearch5.X相关核心知识点必知必会(如下)。
0. ES相关推荐
首先,不要再使用curl,请安装sense(kibana5.x中默认包含sense)
1)ES官方向导
https://www.elastic.co/guide/en/elasticsearch/guide/master/index.html
2)ES官方文档(API相关)
https://www.elastic.co/guide/en/elasticsearch/reference/master/index.html
3)ES资源清单(全)
https://github.com/dzharii/awesome-elasticsearch
4)ES云
https://www.found.no/play
5)ES官方论坛(英文)
https://discuss.elastic.co/
6)ES|stackOverflow论坛地址
https://stackoverflow.com/questions/tagged/elasticsearch
7)ES 性能测试相关(NB)
https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/
1. 查询操作
基本查询有两种语法:左边是一个简单的语法,你不能使用任何选项,
右边是一个扩展。 大多数初学者头痛的DSL来自于:
GET _search
{
"query": {
"match": {
"FIELD": "TEXT"
}
}
}
GET _search
{
"query": {
"match": {
"FIELD": {
"query": "TEXT",
"OPTION": "VALUE"
}
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
1.1 包含高亮、聚合、过滤器的完整的检索示例
GET /_search
{
"query": {
"bool": {
"must": [
{
"match": {
"title": "smith"
}
}
],
"must_not": [
{
"match_phrase": {
"title": "granny smith"
}
}
],
"filter": [
{
"exists": {
"field": "title"
}
}
]
}
},
"aggs": {
"my_agg": {
"terms": {
"field": "user",
"size": 10
}
}
},
"highlight": {
"pre_tags": [
"<em>"
],
"post_tags": [
"</em>"
],
"fields": {
"body": {
"number_of_fragments": 1,
"fragment_size": 20
},
"title": {}
}
},
"size": 20,
"from": 100,
"_source": [
"title",
"id"
],
"sort": [
{
"_id": {
"order": "desc"
}
}
]
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
1.2 普通检索
多字段检索
"multi_match": {
"query": "Elastic",
"fields": ["user.*", "title^3"],
"type": "best_fields"
}
- 1
- 2
- 3
- 4
- 5
bool检索
"bool": {
"must": [],
"must_not": [],
"filter": [],
"should": [],
"minimum_should_match" : 1
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
范围检索
"range": {
"age": {
"gte": 10,
"lte": 20,
"boost": 2
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
1.3 QueryString语法概述
1.3.1 检索所有的_all字段
GET /_search?q=pony
- 1
1.3.2 包含运算符和包含boost精确检索的复杂检索
GET /_search?q=title:(joli OR code) AND author:"Damien Alexandre"^2
- 1
1.3.3 使用通配符和特殊查询进行检索
GET /_search?q=_exists_:title OR title:singl? noneOrAnyChar*cter
- 1
1.3.4 模糊搜素和范围检索
GET /_search?q=title:elastichurch~3 AND date:[2016-01-01 TO 2018-12-31]
- 1
1.3.5 使用 DSL检索(不推荐用于用户搜索):
GET /_search
{
"query": {
"query_string": {
"default_field": "content",
"query": "elastic AND (title:lucene OR title:solr)"
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
2. 索引操作
2.1 创建包含设置和mapping的索引
PUT /my_index_name
{
"settings": {
"number_of_replicas": 1,
"number_of_shards": 3,
"analysis": {},
"refresh_interval": "1s"
},
"mappings": {
"my_type_name": {
"properties": {
"title": {
"type": "text",
"analyzer": "english"
}
}
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
2.2 动态的更新设置
PUT /my_index_name/_settings
{
"index": {
"refresh_interval": "-1",
"number_of_replicas": 0
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
2.3 通过向类型添加字段更新索引
PUT /my_index_name/_mapping/my_type_name
{
"my_type_name": {
"properties": {
"tag": {
"type": "keyword"
}
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
2.4 获取Mapping和设置
GET /my_index_name/_mapping
GET /my_index_name/_settings
- 1
- 2
2.5 创建document
POST /my_index_name/my_type_name
{
"title": "Elastic is funny",
"tag": [
"lucene"
]
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
2.6 创建或更新document
PUT /my_index_name/my_type_name/12abc
{
"title": "Elastic is funny",
"tag": [
"lucene"
]
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
2.7 删除文档
DELETE /my_index_name/my_type_name/12abc
- 1
2.8 打开或关闭索引已节约内存和CPU
POST /my_index_name/_close
POST /my_index_name/_open
- 1
- 2
2.9 移除和创建别名
POST /_aliases
{
"actions": [
{
"remove": {
"index": "my_index_name",
"alias": "foo"
}
},
{
"add": {
"index": "my_index_name",
"alias": "bar",
"filter" : { "term" : { "user" : "damien" } }
}
}
]
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
2.10 列举别名
GET /_aliases
GET /my_index_name/_alias/*
GET /*/_alias/*
GET /*/_alias/foo
- 1
- 2
- 3
- 4
2.11 索引监控和信息
GET /my_index_name/_stats
GET /my_index_name/_segments
GET /my_index_name/_recovery?pretty&human
- 1
- 2
- 3
2.12 索引状态和管理
POST /my_index_name/_cache/clear
POST /my_index_name/_refresh
POST /my_index_name/_flush
POST /my_index_name/_forcemerge
POST /my_index_name/_upgrade
GET /my_index_name/_upgrade?pretty&human
- 1
- 2
- 3
- 4
- 5
- 6
3. 调试和部署
3.1 检索调试
3.1.1 获取query操作到底做了什么?
GET /blog/post/_validate/query?explain
{
"query": {
"match": {
"title": "Smith"
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
3.1.2 获取文档是否匹配?
GET /blog/post/1/_explain
{
"query": {
"match": {
"title": "Smith"
}
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
3.2 分析
3.2.1 测试内容如何在文档中被标记?
GET /blog/_analyze?field=title&text=powerful
- 1
3.2.2 测试分析器输出?
GET /_analyze?analyzer=english&text=powerful
- 1
3.3 集群管理和插件管理
3.3.1 集群和节点信息
GET /_cluster/health?pretty
GET /_cluster/health?wait_for_status=yellow&timeout=50s
GET /_cluster/state
GET /_cluster/stats?human&pretty
GET /_cluster/pending_tasks
GET /_nodes
GET /_nodes/stats
GET /_nodes/nodeId1,nodeId2/stats
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
3.3.2 手动移动分片
索引1的分片移动到索引2
POST /_cluster/reroute
{
"commands": [
{
"move": {
"index": "my_index_name",
"shard": 0,
"from_node": "node1",
"to_node": "node2"
}
},
{
"allocate": {
"index": "my_index_name",
"shard": 1,
"node": "node3"
}
}
]
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
3.3.3 更新设置
动态更新最小节点数。
PUT /_cluster/settings
{
"persistent": {
"discovery.zen.minimum_master_nodes": 3
}
}
PUT /_cluster/settings
{
"transient": {
"discovery.zen.minimum_master_nodes": 2
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
使分片失效,在rolling重启前有效。
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
4.最有用的插件
ES5.X版本已不再支持站点插件,请查看kibana应用或类似Cerebro等其他独立的app做管理。
Cerebro地址: https://github.com/lmenezes/cerebro
Analysis ICU
从Unicode ICU库中添加有用的tokenizer和令牌过滤器。
bin/elasticsearch-plugin install analysis-icu
- 1
AWS Cloud
允许在Amazon云(EC2和S3)中发现和存储。
bin / elasticsearch-plugin install discovery-ec2 bin / elasticsearch-plugin install repository-s3
- 1
Azure Cloud
允许在微软云(EC2和S3)中发现和存储。
bin/elasticsearch-plugin install discovery-azure-classicbin/elasticsearch-plugin install repository-azure
- 1
插件管理:
bin/elasticsearch-plugin install file:///path/to/plugin
bin/elasticsearch-plugin list
bin/elasticsearch-plugin remove [pluginname]
- 1
- 2
- 3
5.其他信息
5.1 如何发现其他插件
RPM: /usr/share/elasticsearch/bin
Debian: /usr/share/elasticsearch/bin
- 1
- 2
5.2 缺省端口
Kibana: http://localhost:5601/.
Elasticsearch: http://localhost:9200/.
- 1
- 2
- 3
5.3 如何设置堆大小
单一Elasticsearch 服务器最大可用内存小于总内存的50%,且小于32GB;
假定你使用Ubuntu/Debian 服务器,你可以通过如下配置修改:
/etc/security/limits.conf
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
- 1
- 2
- 3
- 4
对于Centos服务器,修改配置如下:
/etc/default/elasticsearch (on CentOS/RH: /etc/sysconfig/elasticsearch)
ES_HEAP_SIZE=20g
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
- 1
- 2
- 3
- 4
- 5
5.4 elasticsearch.yml中有用的需要改变的配置
cluster.name: jolicluster
node.name: ${HOSTNAME}
discovery.zen.ping.unicast.hosts: ["front01", "front02"]
discovery.zen.minimum_master_nodes: 2
network.host: _site_
network.bind_host: [_site_, _local_]
plugin.mandatory: analysis-icu
node.data: true
node.master: true
bootstrap.memory_lock: true
action.auto_create_index: +aaa*,-bbb*,+ccc*,-*
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
6. 小结
本文是翻译的 http://elasticsearch-cheatsheet.jolicode.com/#es5。
Cheat sheet是作弊表的意思。可见源作者对文章寄托的意义就是最核心、最关键的ES知识点“小抄”。
我把它理解成ES学习中待翻看的字典。
尤其第0章节的相关链接,都是上乘ES佳作,待进一步深度剖析。