Elasticsearch集群索引生命周期管理

一.需求:

数据量每天几TB甚至几十TB的增长时,索引的生命周期管理显得尤为重要。
Elasticsearch7.0以上已经内置了过期策略,这里不再赘述,这里以Elasticsearch6.x版本为例进行配置和使用。

二.方案:

1.工具 :Elasticsearch索引管理利器——Curator

	1):Github地址:https://github.com/elastic/curator
	2):使用版本注意:
		6.X ES使用 curator 5;
		5.X ES可以使用curator5 或者 curator4 ,具体参考官网:https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html
	
	3):curator 命令行直接使用
1.curator_cli 命令执行
[root@es-prod-0002 elasticsearch-curator]# ./curator_cli --host 192.168.0.180 --port 9200 close --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":1},{"filtertype":"pattern","kind":"prefix","value":"gateway-"}]'
	2021-03-29 16:57:16,007 INFO      Instantiating client object
	2021-03-29 16:57:16,008 INFO      Testing client connectivity
	2021-03-29 16:57:16,010 INFO      Successfully created Elasticsearch client object with provided settings
	2021-03-29 16:57:17,403 INFO      Closing 2 selected indices: ['gateway-error-log-2021-03-27', 'gateway-error-log-2021-03-28']
	2021-03-29 16:57:17,574 INFO      "close" action completed.



[root@es-prod-0002 elasticsearch-curator]# ./curator_cli --host 192.168.0.180 --port 9200  delete_indices  --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":1},{"filtertype":"pattern","kind":"prefix","value":"gateway-"}]'
	2021-03-29 16:58:31,815 INFO      Instantiating client object
	2021-03-29 16:58:31,816 INFO      Testing client connectivity
	2021-03-29 16:58:31,818 INFO      Successfully created Elasticsearch client object with provided settings
	2021-03-29 16:58:33,301 INFO      Deleting 2 selected indices: ['gateway-error-log-2021-03-27', 'gateway-error-log-2021-03-28']
	2021-03-29 16:58:33,301 INFO      ---deleting index gateway-error-log-2021-03-27
	2021-03-29 16:58:33,301 INFO      ---deleting index gateway-error-log-2021-03-28
	2021-03-29 16:58:33,690 INFO      "delete_indices" action completed.
	
	好处:无需配置文件,一行命令即可成功。
	坏处:不能便捷的适应复杂的操作。

4):curator 的yaml脚本

config.yaml:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 192.168.0.180
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 120
  master_only: False
 
logging:
  loglevel: INFO
  logfile: /opt/elasticsearch-curator/log/run.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action.yaml:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 20 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: '^(skywalking).*$'
    - filtertype: age
      source: name
      direction: older
      #timestring: '%Y-%m-%d' 要查看下索引的日期格式
      timestring: '%Y%m%d'
      unit: days
      unit_count: 14

手动执行:
/opt/elasticsearch-curator/curator --config /opt/elasticsearch-curator/config.yml /opt/elasticsearch-curator/action.yml
确认curl -s -XGET ‘http://192.168.0.180:9200/_cluster/health?pretty’ 索引数量是否减少,以及清理删除索引的日志信息如下:

[root@es-prod-0002 log]# pwd
/opt/elasticsearch-curator/log
[root@es-prod-0002 log]# tail -100 run.log 
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_instance_clr_available_completion_port_threads-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_service_relation_client_cpm-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_service_relation_client_side-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_profile_task_log-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_meter_jvm_threads_daemon-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_service_instance_relation_server_resp_time-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_http_server_requests_count-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_endpoint_relation_percentile-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_jvm_memory_max-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_browser_app_page_ajax_error_sum-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_meter_jvm_threads_peak-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_endpoint_relation_sla-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_envoy_parent_connections_used-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_envoy_heap_memory_max_used-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_service_instance_resp_time-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_browser_app_page_error_sum-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_process_files_max-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_meter_jdbc_connections_max-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_jvm_classes_unloaded-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_service_instance_relation_client_call_sla-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_tomcat_sessions_rejected-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_zipkin_span-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_database_access_sla-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_service_relation_server_cpm-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_jvm_threads_daemon-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_meter_jvm_gc_pause_count-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_instance_jvm_old_gc_count-20210408
2021-04-22 14:38:43,411 INFO      ---deleting index skywalking_gray_browser_app_single_version_error_sum-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_resource_error_sum-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_instance_jvm_young_gc_time-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_browser_app_page_unknown_error_sum-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_service_percentile-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_error_log-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_instance_jvm_old_gc_time-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_load_page_avg-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_meter_system_cpu_usage-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_ttl_avg-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_meter_system_cpu_usage-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_instance_clr_cpu-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_js_error_sum-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_browser_error_log-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_single_version_pv-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_res_avg-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_top_n_database_statement-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_endpoint_relation_sla-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_browser_app_page_first_pack_percentile-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_instance_jvm_memory_noheap-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_endpoint_relation_server_side-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_instance_clr_available_completion_port_threads-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_meter_jvm_memory_committed-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_browser_app_page_resource_error_sum-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_envoy_total_connections_used-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_gray_all_percentile-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_browser_app_page_redirect_avg-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_browser_app_page_first_pack_percentile-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_service_apdex-20210408
2021-04-22 14:38:43,412 INFO      ---deleting index skywalking_endpoint_relation_percentile-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_browser_app_page_fmp_avg-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_browser_app_page_dom_analysis_avg-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_meter_system_load_average_1m-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_network_address_alias-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_service_relation_server_resp_time-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_meter_jvm_memory_committed-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_browser_app_page_redirect_avg-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_instance_traffic-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_service_instance_relation_server_call_sla-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_service_instance_relation_server_cpm-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_service_relation_client_cpm-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_browser_app_page_ttl_percentile-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_browser_app_page_fpt_percentile-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_database_access_cpm-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_meter_jvm_gc_pause_count-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_endpoint_relation_resp_time-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_service_instance_relation_server_side-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_browser_app_page_error_rate-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_endpoint_sla-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_instance_clr_heap_memory-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_service_relation_server_call_sla-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_endpoint_sla-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_gray_browser_app_page_trans_avg-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_segment-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_profile_task_segment_snapshot-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_instance_jvm_thread_live_count-20210408
2021-04-22 14:39:02,402 INFO      ---deleting index skywalking_meter_jvm_memory_used-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_service_relation_client_side-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_instance_jvm_old_gc_count-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_browser_app_page_dns_avg-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_endpoint_cpm-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_browser_app_page_unknown_error_sum-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_browser_app_page_res_avg-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_meter_jvm_threads_live-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_browser_app_page_dom_ready_percentile-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_gray_instance_jvm_young_gc_count-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_service_instance_cpm-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_all_percentile-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_endpoint_relation_cpm-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_meter_http_server_requests_duration-20210408
2021-04-22 14:39:02,403 INFO      ---deleting index skywalking_endpoint_traffic-20210408
2021-04-22 14:39:15,420 INFO      Action ID: 1, "delete_indices" completed.
2021-04-22 14:39:15,420 INFO      Job completed.

待确认删除无误后,再crontab 定时任务:
30 3 * * * /opt/elasticsearch-curator/curator --config /opt/elasticsearch-curator/config.yml /opt/elasticsearch-curator/action.yml

2.shell脚本

比较粗暴的形式,直接通过curl 来操作RESTful web进行删除,具体根据自己索引的正则来匹配要删除的索引,以下仅为参考。

#!/bin/bash
#delete 3-30days  index  格式为:skywalking_sdfwe-20200825
for((i=3;i<=30;i++));
do
        echo $i
        LAST_DATA=`date -d "-$i days" "+%Y%m%d"`
        echo $LAST_DATA
        curl -XDELETE http://192.168.0.180:9200/skywalking_*${LAST_DATA}
done

3.总结

curator适用于基于时间或者template其他方式创建的索引,不适合单一索引存储N久历史数据的操作的场景。
ES 7.0及以上都有Kibana自带可视化生命周期管理,具体可以参考ES官网自行查看和使用。
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值