ES 记录

余建新-18588497759

已于 2023-02-17 16:39:07 修改

阅读量3.4k

点赞数

分类专栏： Elasticsearch 文章标签： Elasticsearch

于 2017-06-13 10:40:56 首次发布

本文链接：https://blog.csdn.net/aaronhadoop/article/details/73163025

版权

Elasticsearch 专栏收录该内容

33 篇文章 1 订阅

订阅专栏

`参数`

cluster.routing.allocation.enable 是否分片自动均衡

`ES 常用操作`

curl '0.0.0.0:9200/_cat/health?v'
curl '0.0.0.0:9200/_cat/nodes?v'
curl '0.0.0.0:9200/_cat/indices?v'


curl -XGET '0.0.0.0:9200/index_20220527/_count?pretty'
curl -XGET '0.0.0.0:9200/index_20220527/_search?pretty'

写入数据
curl -XPOST -H "Content-Type: application/json" '0.0.0.0:9200/index_20220527/1?pretty' -d' {
    "field_1" : "value_1",
    "value_2" :  "value_2"
}'

删除索引
curl -XDELETE '0.0.0.0:9200/index_20220527?pretty'

一、DDL

## 查看 mapping
curl http://ES_IP:ES_PORT/INDEX_NAME-INDEX_DATE/_mappings?pretty

## 创建索引
curl -H 'Content-Type: application/json' -XPUT "http://ES_IP:ES_PORT/INDEX_NAME?pretty" -d '
{
    "settings":{
        "number_of_shards":40,
        "number_of_replicas":0,
        "index.routing.allocation.total_shards_per_node":5
    },
    "mappings":{
        "collect_data_base_info":{
            "properties":{
                "createTime":{
                    "type":"date",
                    "format":"yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
                },
                "appSystem":{
                    "type":"keyword"
                },
                "channelId":{
                    "type":"integer"
                }
            }
        }
    }
}'

## 创建别名
curl -XPOST "http://${ES_IP}:${ES_PORT}/_aliases" -d '{
    "actions":[
        {
            "add":{
                "index":"${INDEX_NAME}",
                "alias":"${ALIAS_NAME}"
            }
        }
    ]
}'

在这里插入图片描述

新增字段
在这里插入图片描述

新增字段的 url
uid_list_202203/uid_list/_mapping

{
    "properties":{
        "Ffee_tps_id":{
            "type":"keyword"
        }
    }
}

二、DML

## 查看最新 5 条记录
{
    "sort":[
        {
            "FIELD_NAME":{
                "order":"desc"
            }
        }
    ],
    "from":0,
    "size":5
}


## 某个字段的匹配查询
{
    "query":{
        "term":{
            "app_name":"fm.fenqile.com"
        }
    },
    "from":0,
    "size":2
}


## 某个字段的模糊查询
{
    "query": {
        "wildcard": {
            "headers.fileAbsolutePath": {
                "value": "*cia_pettyloan_sec_server_logs*"
            }
        }
    }
}

## GROUP BY field, COUNT(1), ORDER BY DESC, LIMIT 20
{
    "aggs":{
        "cnt":{
            "terms":{
                "field":"app_name",
                "order":{
                    "_count":"desc"
                },
                "size": 20
            }
        }
    }
}
参考，https://www.cnblogs.com/duanxz/p/6528161.html

## OR 查询
SELECT * FROM indexName WHERE host_ip="ip1" OR host_ip="ip2" ORDER BY host_ip LIMIT 0, 3

{
    "query":{
        "bool":{
            "should":[
                {
                    "term":{
                        "host_ip":"ip1"
                    }
                },
                {
                    "term":{
                        "host_ip":"ip2"
                    }
                }
            ]
        }
    },
    "sort":[
        {
            "host_ip":{
                "order":"desc"
            }
        }
    ],
    "from":0,
    "size":3
}





需求：
按FIELD_NAME 统计count，取top100（GROUP BY FIELD_NAME LIMIT 100）

报错信息：
{
  "error" : {
    "root_cause" : [
      {
        "type" : "remote_transport_exception",
        "reason" : "[ES_NODE_NAME][ES_NODE_IP:ES_NODE_PORT][indices:admin/mapping/put]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Mapper for [FIELD_NAME] conflicts with existing mapping in other types:\n[mapper [FIELD_NAME] is used by multiple types. Set update_all_types to true to update [fielddata] across all types.]"
  },
  "status" : 400
}

解决：
curl -XPUT 'http://ES_IP:ES_PORT/INDEX_NAME-INDEX_DATE/_mapping/storm_worker?update_all_types&pretty' -H 'Content-Type: application/json' -d '{
  "properties": {
    "FIELD_NAME": {
      "type":     "text",
      "fielddata": true
    }
  }
}'

然后统计：
curl -XPOST 'http://ES_IP:ES_PORT/INDEX_NAME-INDEX_DATE/_search?pretty' -d '{
    "aggs":{
        "FIELD_NAME":{
            "terms":{
                "field":"FIELD_NAME",
                "order":[
                    {
                        "_count":"desc"
                    }
                ],
                "size":100
            },
            "aggs":{

            }
        }
    },
    "size":0
}' > INDEX_NAME-INDEX_DATE_groupby_FIELD_NAME_top100.txt

参考
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-put-mapping.html

三、其他 ES 操作

## 查看索引列表
curl "http://172.19.105.85:9200/_cat/indices?v"

## 写入数据
curl -XPOST -H "Content-Type: application/json" "http://${ES_NODE_IP}:9200/lx_20211229/type_lx?pretty" -d '
{
    "id": "1",
    "field_1": "value_1"
} '

## 查询数据
curl "http://${ES_NODE_IP}:9200/lx_20211229/_search?pretty"


## 创建别名 BEGIN

1、删除原索引
curl -XDELETE ES_IP:ES_PORT/af_collect_data_2021q4

2、创建别名，保留对外名称不变，指向新索引
curl -H "Content-Type: application/json" -XPOST 10.9.53.10:9200/_aliases -d '{
    "actions":[
        {
            "add":{
                "index":"af_collect_data_2021q4_new",
                "alias":"af_collect_data_2021q4"
            }
        }
    ]
}'

3、查看已经存在的别名
curl http://ES_IP:ES_PORT/_cat/aliases?v

## 创建别名 END

默认 index template

{
    "order":0,
    "template":"*",
    "settings":{
        "index":{
            "number_of_shards":"1",
            "number_of_replicas":"1",
            "refresh_interval":"60s"
        }
    },
    "mappings":{

    },
    "aliases":{

    }
}

写入 ES 异常，修改 read_only_allow_delete 属性

在这里插入图片描述

index.blocks.read_only_allow_delete: false
cluster.blocks.read_only_allow_delete: false


curl -u "elastic:PASSWORD" -XPUT -H "Content-Type: application/json" IP:9291/_all/_settings -d '
{
    "index.blocks.read_only_allow_delete":false
}
'

curl -u "elastic:PASSWORD" -XPUT -H "Content-Type: application/json" IP:9291/_cluster/_settings?pretty" -d ' {
    "persistent":{
        "cluster":{
            "blocks":{
                "read_only_allow_delete": false
            }
        }
    }
}'

2. ES 创建 mapping 模板

这里写图片描述
转自祝威廉ES

ES 优化

ES 记录

1. 问题记录
这些属性在ES5.x中被移除了
discovery.zen.ping.multicast.enabled: false
index.number_of_shards: 20
index.number_of_replicas: 1


报错日志
*************************************************************************************
Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes 
configuration like the elasticsearch.yaml, in system properties or command line 
arguments.In order to upgrade all indices the settings must be updated via the 
/${index}/_settings API. Unless all settings are dynamic all indices must be closed 
in order to apply the upgradeIndices created in the future should use index templates 
to set default values. 

Please ensure all required values are updated on all indices by executing: 

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_replicas" : "1",
  "index.number_of_shards" : "20"
}'
*************************************************************************************

[2017-06-12T17:06:09,942][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-xxx-07-node3] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.0.jar:5.4.0]
Caused by: java.lang.IllegalArgumentException: node settings must not contain any index level settings
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.node.Node.<init>(Node.java:342) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0]
        ... 6 more

ES 优化

修改vm.max_map_count

问题：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决：
vi /etc/sysctl.conf 
添加下面配置：
vm.max_map_count=655360

命令行执行sysctl -p

ES 优化

ES 压缩

curl -XPOST 'es_ip:8080/index_in_es5_1/1?pretty' -d' {
    "first_name" : "John22",
    "last_name" :  "Smith",
    "age" :        25,
    "about" :      "I love to go rock climbing",
    "interests": [ "sports", "music" ]
}'

curl -XPUT "es_ip:8080/index_in_es5_1/_settings" -d ' {
    "number_of_replicas" : 1
}'

curl -XPUT "es_ip:8080/test_index/_settings?pretty" -d '{
    "index.codec" : "best_compression"
}'

curl -XPUT "es_ip:8080/test_index/_settings?pretty" -d ' {"index":{"codec":"best_compression"} }'


参考
https://stackoverflow.com/questions/38343499/compression-to-elasticsearch-indexes
https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0
https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch