ElasticSearch 知识点整理

目录

一、前文

二、概念

2.1 集群

2.2 读写请求

2.3 优化

2.3.1 部署调优

2.3.2 配置调优

2.3.3 DSL调优

2.4 其他

三、实战

3.1 Mysql - Es 映射

3.2 Hive - Es映射

---

一、前文

             目前公司有3套ES集群环境：直播业务集群(6节点)，论坛业务集群(3节点)，日志集群(3节点)，应用场景分别为线上直播及订单商品等业务查询、大数据报表数据查询、日志监控等。其中使用Filebeat，Logstash，SparkStreaming 等接入数据，Kibana 负责监控。

二、概念

2.1 集群

2.2 读写请求

 

注：该图为其他博主所画，如有侵权请联系删除。

2.3 优化

2.3.1 部署调优

2.3.2 配置调优

2.3.3 DSL调优

 

2.4 其他

三、实战

3.1 Mysql - Es 映射

 

优化索引：

PUT xxxxxx/
{
  "settings": {
    "index.search.slowlog.threshold.query.warn": "4s",
    "index.search.slowlog.threshold.query.info": "2s",
    "index.search.slowlog.threshold.query.debug": "1s",
    "index.search.slowlog.threshold.query.trace": "100ms",
    "index.search.slowlog.threshold.fetch.warn": "2s",
    "index.search.slowlog.threshold.fetch.info": "1s",
    "index.search.slowlog.threshold.fetch.debug": "-1",
    "index.search.slowlog.threshold.fetch.trace": "-1",
    "index.search.slowlog.level": "info",
    "refresh_interval": "10s",
    "index.merge.policy.floor_segment": "50m",
    "index.merge.policy.max_merged_segment": "2gb",
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic_date_formats": [
      "yyyy-MM-dd HH:mm:ss"
    ],
    "properties": {
      "Id": {
        "type": "keyword",
        "doc_values": false
      },
      "userId": {
        "type": "keyword",
        "doc_values": false
      },
      "Name": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      },
      "mininsnsId": {
        "type": "keyword",
        "doc_values": false
      },
      "TplName": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "Descript": {
        "type": "text",
        "index": false,
        "doc_values": false
      },
      "state": {
        "type": "keyword",
        "doc_values": false
      },
      "MyQRCode": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "IsOpenNew": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "Levels": {
        "type": "keyword",
        "doc_values": false
      },
      "area": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "isOpenplat": {
        "type": "keyword",
        "doc_values": false
      },
      "province": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "city": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "CustomerType": {
        "type": "keyword",
        "doc_values": false
      },
      "addtime": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
      "Setting": {
        "type": "keyword",
        "index": false,
        "doc_values": false
      },
      "applyId": {
        "type": "keyword",
        "doc_values": false
      },
      "isFlowPay": {
        "type": "keyword",
        "doc_values": false
      }
    }
  }
}

3.2 Hive - Es映射

drop table ads.es_cdnlog;
create table ads.es_cdnlog(
createdate string comment '流量日期',
zbid bigint comment '直播id',
stream string comment '流量id',
livetraffic bigint comment '直播流量',
backtraffic bigint comment '回放流量'
)
stored	by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
tblproperties(
'es.nodes' = 'es_01:9200,es_02:9200',
'es.resource' = 'cdn_traffic_day',
'es.mapping.date.rich'='false',
'es.date.format'='yyyy-MM-dd',
'es.net.http.auth.user'='elastic','es.net.http.auth.pass'='123456',
'es.mapping.names' = '
createdate:createDate,
zbid:zbId,
stream:streamId,
livetraffic:liveTrafficSum,
backtraffic:playbackTrafficSum');

注：在Hive 放入官网找到的 elasticsearch-for-apache-hadoop.jar 绑定即可

<property>
    <name>hive.reloadable.aux.jars.path</name>
    <value>/home/hadoopjar/</value>
</property>

3.3 重建索引

POST _reindex                   
{
  "source": {
    "index": "order_transaction_v1"
  },
  "dest": {
    "index": "order_transaction_v2"
  }
}

POST /_aliases
{
  "actions": [
    {
      "remove": {
        "index": "order_transaction_v1",
        "alias": "order_transaction_record"
      }
    },
    {
      "add": {
        "index": "order_transaction_v2",
        "alias": "order_transaction_record"
      }
    }
  ]
}

3.4 Kibana 监控等