08、es 进一步了解___e_性能优化_

最新推荐文章于 2023-07-10 18:00:38 发布

VIP文章春光静好

最新推荐文章于 2023-07-10 18:00:38 发布

阅读量554

点赞数

分类专栏： es 文章标签： elasticsearch 大数据搜索引擎

本文链接：https://blog.csdn.net/ITresource/article/details/104347980

版权

1)存储设备

磁盘在现代服务器上通常都是瓶颈。Elasticsearch 重度使用磁盘，你的磁盘能处理的吞吐量越大，你的节点就越稳定。

这里有一些优化磁盘 I/O 的技巧：

使用 SSD。就像其他地方提过的，他们比机械磁盘优秀多了。
使用 RAID 0。条带化 RAID 会提高磁盘 I/O，代价显然就是当一块硬盘故障时整个就故障了。不要使用镜像或者奇偶校验 RAID 因为副本已经提供了这个功能。
另外，使用多块硬盘，并允许 Elasticsearch 通过多个 path.data 目录配置把数据条带化分配到它们上面。
不要使用远程挂载的存储，比如 NFS 或者 SMB/CIFS。这个引入的延迟对性能来说完全是背道而驰的。

如果你用的是 EC2，当心 EBS。即便是基于 SSD 的 EBS，通常也比本地实例的存储要慢。

2)_内部索引优化

在这里插入图片描述

Elasticsearch 为了能快速找到某个 Term，先将所有的 Term 排个序，然后根据二分法查找 Term，时间复杂度为 logN，就像通过字典查找一样，这就是 Term Dictionary。

现在再看起来，似乎和传统数据库通过 B-Tree 的方式类似。但是如果 Term 太多，Term Dictionary 也会很大，放内存不现实，于是有了 Term Index。

就像字典里的索引页一样，A 开头的有哪些 Term，分别在哪页，可以理解 Term Index是一棵树。

这棵树不会包含所有的 Term，它包含的是 Term 的一些前缀。通过 Term Index 可以快速地定位到 Term Dictionary 的某个 Offset，然后从这个位置再往后顺序查找。

在内存中用 FST 方式压缩 Term Index，FST 以字节的方式存储所有的 Term，这种压缩方式可以有效的缩减存储空间，使得 Term Index 足以放进内存，但这种方式也会导致查找时需要更多的 CPU 资源。

对于存储在磁盘上的倒排表同样也采用了压缩技术减少存储所占用的空间。

3)_调整配置参数

调整配置参数建议如下：

给每个文档指定有序的具有压缩良好的序列模式 ID，避免随机的 UUID-4 这样的 ID，这样的 ID 压缩比很低，会明显拖慢 Lucene
对于那些不需要聚合和排序的索引字段禁用 Doc values。Doc Values 是有序的基于 document=>field value 的映射列表。
不需要做模糊检索的字段使用 Keyword 类型代替 Text 类型，这样可以避免在建立索引前对这些文本进行分词。
如果你的搜索结果不需要近实时的准确度，考虑把每个索引的 index.refresh_interval 改到 30s 。
如果你是在做大批量导入，导入期间你可以通过设置这个值为 -1 关掉刷新，还可以通过设置 index.number_of_replicas: 0 关闭副本。别忘记在完工的时候重新开启它。
避免深度分页查询建议使用 Scroll 进行分页查询。普通分页查询时，会创建一个 from+size 的空优先队列，每个分片会返from+size 条数据，默认只包含文档 ID 和得分 Score 给协调节点。如果有 N 个分片，则协调节点再对（from+size）×n 条数据进行二次排序，然后选择需要被取回的文档。当 from 很大时，排序过程会变得很沉重，占用 CPU 资源严重
减少映射字段，只提供需要检索，聚合或排序的字段。其他字段可存在其他存储设备上，例如 Hbase，在 ES 中得到结果后再去 Hbase 查询这些字段。
创建索引和查询时指定路由 Routing 值，这样可以精确到具体的分片查询，提升查询效率。路由的选择需要注意数据的分布均衡。

4)_JVM 调优

JVM 调优建议如下：

确保堆内存最小值（ Xms ）与最大值（ Xmx ）的大小是相同的，防止程序在运行时改变堆内存大小。
Elasticsearch 默认安装后设置的堆内存是 1GB。可通过 …/config/jvm.option 文件进行配置，但是最好不要超过物理内存的50%和超过 32GB。
GC 默认采用 CMS 的方式，并发但是有 STW 的问题，可以考虑使用 G1 收集器。
ES 非常依赖文件系统缓存（Filesystem Cache），快速搜索。一般来说，应该至少确保物理上有一半的可用内存分配到文件系统缓存。

JVM 调优建议如下：
确保堆内存最小值（ Xms ）与最大值（ Xmx ）的大小是相同的，防止程序在运行时改变堆内存大小。
Elasticsearch 默认安装后设置的堆内存是 1GB。可通过 ../config/jvm.option 文件进行配置，但是最好不要超过物理内存的50%和超过 32GB。

GC 默认采用 CMS 的方式，并发但是有 STW 的问题，可以考虑使用 G1 收集器。

ES 非常依赖文件系统缓存（Filesystem Cache），快速搜索。一般来说，应该至少确保物理上有一半的可用内存分配到文件系统缓存。

{
   
  "_nodes": {
   
    "total": 4,
    "successful": 4,
    "failed": 0
  },
  "cluster_name": "order",
  "nodes": {
   
    "7RcuoaZGSmKf3y6sQkphDA": {
   
      "name": "es3",
      "transport_address": "IP1:9300",
      "host": "IP1",
      "ip": "IP1",
      "version": "6.2.1",
      "build_hash": "7299dc3",
      "total_indexing_buffer": 1932735283,
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "attributes": {
   
        "ml.machine_memory": "33566806016",
        "ml.max_open_jobs": "20",
        "ml.enabled": "true"
      },
      "settings": {
   
        "cluster": {
   
          "name": "order"
        },
        "node": {
   
          "max_local_storage_nodes": "1",
          "name": "es3",
          "attr": {
   
            "ml": {
   
              "machine_memory": "33566806016",
              "max_open_jobs": "20",
              "enabled": "true"
            }
          },
          "data": "true",
          "ingest": "true",
          "master": "true"
        },
        "path": {
   
          "data": [
            "/data1/elasticsearch/data/"
          ],
          "logs": "/data/elasticsearch/logs",
          "home": "/data/elasticsearch"
        },
        "discovery": {
   
          "zen": {
   
            "fd": {
   
              "ping_interval": "10s",
              "ping_retries": "10",
              "ping_timeout": "60s"
            },
            "ping": {
   
              "unicast": {
   
                "hosts": [
                  "IP2",
                  "IP3",
                  "IP1",
                  "IP4",
                  "IP5"
                ]
              }
            }
          }
        },
        "client": {
   
          "type": "node"
        },
        "http": {
   
          "type": {
   
            "default": "netty4"
          },
          "port": "9200",
          "cors": {
   
            "allow-origin": "*",
            "allow-headers": "X-Requested-With, X-Auth-Token, Content-Type, Content-Length, Authorization",
            "allow-methods": "OPTIONS, HEAD, GET, POST, PUT, DELETE",
            "enabled": "true"
          }
        },
        "bootstrap": {
   
          "memory_lock": "true"
        },
        "transport": {
   
          "type": {
   
            "default": "netty4"
          }
        },
        "xpack": {
   
          "security": {
   
            "enabled": "false"
          }
        },
        "network": {
   
          "host": "0.0.0.0"
        }
      },
      "os": {
   
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "arch": "amd64",
        "version": "3.10.0-693.el7.x86_64",
        "available_processors": 16,
        "allocated_processors": 16
      },
      "process": {
   
        "refresh_interval_in_millis": 1000,
        "id": 25520,
        "mlockall": true
      },
      "jvm": {
   
        "pid": 25520,
        "version": "1.8.0_91",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.91-b14",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1552970296340,
        "mem": {
   
          "heap_init_in_bytes": 19327352832,
          "heap_max_in_bytes": 19327352832,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 19327352832
        },
        "gc_collectors": [
          "G1 Young Generation",
          "G1 Old Generation"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "G1 Eden Space",
          "G1 Survivor Space",
          "G1 Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true",
        "input_arguments": [
          "-Xms18g",
          "-Xmx18g",
          "-Xmn9g",
          "-XX:+UseG1GC",
          "-XX:MaxGCPauseMillis=200",
          "-XX:+DisableExplicitGC",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-Djdk.io.permissionsUseCanonicalPath=true",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Dlog4j.skipJansi=true",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-Des.path.home=/data/elasticsearch",
          "-Des.path.conf=/data/elasticsearch/config"
        ]
      },
      "thread_pool": {
   
        "watcher": {
   
          "type": "fixed",
          "min": 50,
          "max": 50,
          "queue_size": 1000
        },
        "force_merge": {
   
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "ml_datafeed": {
   
          "type": "fixed",
          "min": 20,
          "max": 20,
          "queue_size": 200
        },
        "fetch_shard_started": {
   
          "type": "scaling",
          "min": 1,
          "max": 32,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
   
          "type": "fixed",
          "min": 8,
          "max": 8,
          "queue_size": -1
        },
        "ml_autodetect": {
   
          "type": "fixed",
          "min": 80,
          "max": 80,
          "queue_size": 80
        },
        "index": {
   
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 200
        },
        "refresh": {
   
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "generic": {
   
          "type": "scaling",
          "min": 4,
          "max": 128,
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
   
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
   
          "type": "fixed_auto_queue_size",
          "min": 25,
          "max": 25,
          "queue_size": 1000
        },
        "flush": {
   
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
   
          "type": "scaling",
          "min": 1,
          "max": 32,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
   
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "ml_utility": {
   
          "type": "fixed",
          "min": 80,
          "max": 80,
          "queue_size": 500
        },
        "get": {
   
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 1000
        },
        "bulk": {
   
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 200
        },
        "snapshot": {
   
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
   
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "192.168.50.15:9300",
        "profiles": {
   }
      },
      "http": {
   
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "192.168.50.15:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
   
          "name": "analysis-ik",
          "version": "6.2.1",
          "description": "IK Analyzer for Elasticsearch",
          "classname": "org.elasticsearch.plugin.analysis.ik.AnalysisIkPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "x-pack-core",
          "version": "6.2.1",
          "description": "Elasticsearch Expanded Pack Plugin - Core",
          "classname": "org.elasticsearch.xpack.core.XPackPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "x-pack-deprecation",
          "version": "6.2.1",
          "description": "Elasticsearch Expanded Pack Plugin - Deprecation",
          "classname": "org.elasticsearch.xpack.deprecation.Deprecation",
          "extended_plugins": [
            "x-pack-core"
          ],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "x-pack-graph",
          "version": "6.2.1",
          "description": "Elasticsearch Expanded Pack Plugin - Graph",
          "classname": "org.elasticsearch.xpack.graph.Graph",
          "extended_plugins": [
            "x-pack-core"
          ],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "x-pack-logstash",
          "version": "6.2.1",
          "description": "Elasticsearch Expanded Pack Plugin - Logstash",
          "classname": "org.elasticsearch.xpack.logstash.Logstash",
          "extended_plugins": [
            "x-pack-core"
          ],
          "has_native_controller": false,
          "requires_keystore": false
        }
      ],
      "modules": [
        {
   
          "name": "aggs-matrix-stats",
          "version": "6.2.1",
          "description": "Adds aggregations whose input are a list of numeric fields and output includes a matrix.",
          "classname": "org.elasticsearch.search.aggregations.matrix.MatrixAggregationPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "analysis-common",
          "version": "6.2.1",
          "description": """Adds "built in" analyzers to Elasticsearch.""",
          "classname": "org.elasticsearch.analysis.common.CommonAnalysisPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "ingest-common",
          "version": "6.2.1",
          "description": "Module for ingest processors that do not require additional security permissions or have large dependencies and resources",
          "classname": "org.elasticsearch.ingest.common.IngestCommonPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "lang-expression",
          "version": "6.2.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "lang-mustache",
          "version": "6.2.1",
          "description": "Mustache scripting integration for Elasticsearch",
          "classname": "org.elasticsearch.script.mustache.MustachePlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "lang-painless",
          "version": "6.2.1",
          "description": "An easy, safe and fast scripting language for Elasticsearch",
          "classname": "org.elasticsearch.painless.PainlessPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "mapper-extras",
          "version": "6.2.1",
          "description": "Adds advanced field mappers",
          "classname": "org.elasticsearch.index.mapper.MapperExtrasPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "parent-join",
          "version": "6.2.1",
          "description": "This module adds the support parent-child queries and aggregations",
          "classname": "org.elasticsearch.join.ParentJoinPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {
   
          "name": "percolator",
          "version": "6.2.1",
          "description": "Percolator module adds capability to index queries and query these queries by specifying documents",
          "classname": "org.elasticsearch.percolator.PercolatorPlugin",
          "extended_plugins": [],
          "has_native_controller": false,
          "requires_keystore": false
        },
        {

最低0.47元/天解锁文章

春光静好

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
08、es 进一步了解___e_性能优化_

2)_内部索引优化Elasticsearch 为了能快速找到某个 Term，先将所有的 Term 排个序，然后根据二分法查找 Term，时间复杂度为 logN，就像通过字典查找一样，这就是 Term Dictionary。现在再看起来，似乎和传统数据库通过 B-Tree 的方式类似。但是如果 Term 太多，Term Dictionary 也会很大，放内存不现实，于是有了 Term Inde...
复制链接

扫一扫