elasticsearch部分常用操作

注意事项

centOS6还要在elasticsearch.yml里加这个,7不用加
问题原因：因为Centos6不支持SecComp，而ES5.2.1默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

单机版本配置

增加修改

cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
#指定单节点类型
discovery.type: single-node

集群搭建7.4.1版本,配置

3台机器组成一个集群，分别为：a,b,c
a:
编辑a的config/elasticsearch.yml配置文件,修改后如下

# ======================== Elasticsearch Configuration #=========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster #—————————————————
#
# Use a descriptive name for your cluster:
#集群名称
cluster.name: my-application
#
# ------------------------------------ Node ##
#
# Use a descriptive name for the node:
#确定master
node.master: true
#节点名称
node.name: node-1
#
#discovery.zen.minimum_master_nodes: 3
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths #
#
# Path to directory where to store the data (separate multiple locations by comma):
#es数据存放位置，需要手动创建目录和赋予权限
path.data: /opt/soft/data
#
# Path to log files:
#
#es日志存放位置，需要手动创建目录和赋予权限
path.logs: /opt/soft/log
#
# ----------------------------------- Memory #
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network #
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#允许自身各种ip访问
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#对外服务端口
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery #
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#指定集群里的所有节点，9300是集群间相互通信的端口
discovery.seed_hosts:  ["10.209.5.87:9300","10.209.5.88:9300","10.209.5.89:9300"]
#discovery.zen.ping.unicast.hosts: ["10.209.5.79","10.209.5.80","10.209.5.78"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#集群启动指定的可选举的master节点
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway #
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various #—————————————————
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#这两行允许跨域
http.cors.enabled: true
http.cors.allow-origin: "*"
#reindex同步数据，数据迁移需要的其他机器的白名单，不然不能使用reindex,这表示当前节点可以#从以下白名单节点获取数据，通常是其他集群的节点
reindex.remote.whitelist: ["10.209.5.84:9200","10.209.5.78:9200","10.209.1.48:9200","10.209.1.35:5200","10.47.187.45:5200","10.47.195.38:5200"]
#指定冷归档数据的存放位置目录，冷归档的数据可以压缩文件夹后剪切移到其他机器，目录需要手#动创建并赋予权限
path.repo: ["/opt/soft/es_backups/backups", "/opt/soft/es_backups/longterm_backups"]

b:
机器的elasticsearch.yml
其他一样，修改
#注释
#node.master: true
#节点名称
node.name: node-2

c:
机器的elasticsearch.yml
其他一样，修改
#注释
#node.master: true
#节点名称
node.name: node-3

修改每一台机器的内存大小参数（64g为例）
修改config/jvm.options文件,最大不能超过31g,最好不超过整个机器的内存50%
-Xms30g
-Xmx30g

可安装ik分词器
需要指定版本

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.4.1/elasticsearch-analysis-ik-7.4.1.zip

linux优化

关闭交换分区，防止内存置换降低性能

swapoff -a

vim /etc/security/limits.conf

#文尾添加
* soft nofile 65535
* hard nofile 131072
* soft nproc 4096
* hard nproc 4096

vim /etc/sysctl.conf

vm.max_map_count=262145 

#刷新配置

sysctl -p 	

es不允许root启动
#增加用户

useradd esuser　

#切换用户

su esuser 

启动命令：
一定要检查防火墙是否开放9200，9300端口
在解压目录执行命令

./bin/elasticsearch -d

索引映射创建，优化

创建索引es_persist_3

创建索引 es_persist_3
url

put http://ip:port/es_persist_3

json

{
  "settings": {
    "number_of_shards": "12",
    "number_of_replicas": "1",
    "index.translog.durability": "async",
    "index.translog.sync_interval": "60s",
    "index.translog.flush_threshold_size": "1024mb"
  }
}

创建映射mapping es_persist_3

创建mapping es_persist_3
url

post http://ip:port/es_persist_3/_mapping

json

{
  "properties": {
    "servCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "httpMethod": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "type": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "servVersionProxyType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionStack": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionTime": {
      "type": "date"
    },
    "@version": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "host": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "pAppName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "id": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "receiveSize": {
      "type": "long"
    },
    "authType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "externalTime": {
      "type": "long"
    },
    "cAppName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "returnSize": {
      "type": "long"
    },
    "authCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "statusDesc": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "platformTime": {
      "type": "long"
    },
    "servName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "componentPort": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "esbId": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "responseSize": {
      "type": "long"
    },
    "message": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "logTime": {
      "type": "date"
    },
    "tags": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "receiveTime": {
      "type": "long"
    },
    "@timestamp": {
      "type": "date"
    },
    "messageList": {
      "properties": {
        "sizeX": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "serialNumber": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "header": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "time": {
          "type": "long"
        },
        "body": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "url": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "componentHost": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "cAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "fromIp": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "complete": {
      "type": "boolean"
    },
    "requestSize": {
      "type": "long"
    },
    "logtime": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "callTime": {
      "type": "long"
    },
    "pAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "statusCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    }
  }
}

创建索引 es_persist_4

url

put http://ip:port/es_persist_34

json

{
  "settings": {
    "number_of_shards": "2",
    "number_of_replicas": "1",
    "index.translog.durability": "async",
    "index.translog.sync_interval": "30s",
    "index.translog.flush_threshold_size": "248mb"
  }
}

创建mapping es_persist_4

url

post http://ip:port/es_persist_4/_mapping

json

{
  "properties": {
    "servCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "componentHost": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionCount": {
      "type": "long"
    },
    "sumCallTime": {
      "type": "long"
    },
    "maxCallTime": {
      "type": "long"
    },
    "cAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "minCallTime": {
      "type": "long"
    },
    "startTime": {
      "type": "long"
    },
    "endTime": {
      "type": "long"
    },
    "sumFlowSize": {
      "type": "long"
    },
    "totalCount": {
      "type": "long"
    },
    "servVersionProxyType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    }
  }
}

es的常用指令

删除指定索引，从物理上整个索引的数据删除
url

delete http://ip:port/指定的索引名称

关闭索引，依然占着硬盘，关闭后不可进行io读写
url

post http://ip:port/指定的索引名称/_close

打开索引，占着硬盘，打开后可进行io读写，正常使用
url

post http://ip:port/指定的索引名称/_open

跨集群数据迁移

reindex迁移

b集群请求获取a集群的数据到b集群里,(b集群配置文件需要加上a集群的白名单，见集群安装配置文件)
query可以指定想要的数据，下面是获取指定月份时间段的数据，去掉则是全部数据
“version_type”: "internal"代表覆盖替换冲突的id相同的数据
size是批量条数，太大可能会报错，太小执行较慢
wait_for_completion=false后台异步操作

POST http://bip:bport/_reindex?wait_for_completion=false

{
  "source": {
    "index": "a的索引",
    "remote": {
      "host": "http://aip:aport"
    },
    "size": 1000,
    "query": {
      "range": {
        "receiveTime": {
          "gte": 1635696000000,
          "lt": 1638287999000
        }
      }
    }
  },
  "dest": {
    "index": "b的索引",
    "version_type": "internal"
  }
}

reindex取消命令

reindex执行没结束不想再执行了，成功迁移复制过去的数据依然保留，后续未完成的不再继续

POST _tasks/node_id:task_id/_cancel

reindex查看进度(可以看到node_id:task_id,任务数等)

GET _tasks?detailed=true&actions=*reindex  

如果迁移自动停止了或者成功了就看不到进度，在数据预览里可以看到.tasks这个索引，是自动生成存储task任务的，查看详情，如果没有报错则成功迁移，报错几乎原因都是数据太大，所以需要调小size并重试

冷数据备份快照到磁盘

可压缩成压缩包剪切到其他机器，减少本集群的硬盘占用，需要在yml指定目录，见安装集群配置文件elasticsearch.yml，目录建议放在硬盘空间大的目录，防止快照文件过大导致失败，
而且指定的目录要有权限
path.repo: [“/opt/soft/es_backups/backups”, “/opt/soft/es_backups/longterm_backups”]

1.创建仓库：创建一个名称为my_backup的仓库

PUT /_snapshot/my_backup

{
  "type": "fs",
  "settings": {
    "location": "/opt/soft/es_backups/backups/my_backup"
  }
}

如果生成快照会在/opt/soft/es_backups/backups下生成仓库my_backup目录

2.往指定仓库my_backup保存指定索引b_index快照数据,创建一个名为 snapshot_1 的快照
wait_for_completion=true可设置同步异步，多个索引用逗号隔开

put /_snapshot/my_backup/snapshot_1?wait_for_completion=true

{
  "indices": "b_index"
}

也可以对snapshot_1 的快照增量同步
直接原命令再创建一个快照 snapshot_2 就可以了

3.数据迁出
打包仓库my_backup,存储不够，直接把文件夹复制走也行

tar -zcvf my_backup.tar.gz my_backup 

可以把压缩文件剪切转储其他机器,然后删除这个文件夹
做的更彻底些，可以删除这个快照和仓库，每次创建快照都重开一个仓库，建议用日期后缀

delete /_snapshot/my_backup/snapshot_1
delete /_snapshot/my_backup

数据包文件迁出后，可以把这个索引删除，解除对硬盘的占用

4.数据恢复
如果是新机器上恢复，新机器也要有仓库目录
把快照文件传过来，在快照目录下解压，解压的文件会生成目录my_backup和新机器指定的目录名称需要一致
执行
创建仓库

PUT /_snapshot/my_backup

{
  "type": "fs",
  "settings": {
    "location": "/opt/soft/es_backups/backups/my_backup"
  }
}

读取数据并创建索引

POST /_snapshot/my_backup/snapshot_1/_restore?wait_for_completion=true

{
  "indices": "index_1"
}

冷数据备份到其他地方，可选

{
  "type": "url",
  "settings": {
    "url": "file:/mount/backups/my_fs_backup_location"
  }
}

Url 参数支持以下协议:

file
ftp
http
https
jar
使用 ftp、 http 或 https 协议的 url 必须使用 repositories.url.allowed _ url 设置显式地被允许
支持通配符(*)，需要配置参数，上面的配置文件未添加该配置

repositories.url.allowed_urls: ["http://www.example.org/root/*", "https://*.mydomain.com/*?*#*"]

集群安全重启

关闭前执行

PUT _cluster/settings

{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}

重启过后执行恢复
请确保每个节点都启动了，不然会造成主分片负载不均衡

{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}

分片移动

若发现主分片负载不均衡，可手动移动分片
将node-1的2号分片移动到node-3

post _cluster/reroute

{
  "commands": [
    {
      "move": {
        "index": "aindex",
        "shard": 2,
        "from_node": "node-1",
        "to_node": "node-3"
      }
    }
  ]
}

如果重启有部分分片无法恢复，执行可重试

POST /_cluster/reroute?retry_failed=true