Elasticsearch 7.8 Docker 集群搭建

最新推荐文章于 2024-04-26 17:22:53 发布

Doraman.Liu

最新推荐文章于 2024-04-26 17:22:53 发布

阅读量1.3k

点赞数

分类专栏：运维开发文章标签： elasticsearch docker 大数据

本文链接：https://blog.csdn.net/m0_37686374/article/details/107748430

版权

运维开发专栏收录该内容

2 篇文章 0 订阅

订阅专栏

Elasticsearch 7.8 Docker 集群搭建

一、安装 docker 环境

1. 更新 centos

# yum update

2. 卸载旧版本 docker

# yum remove docker docker-common docker-selinux docker-engine

3. 安装docker的驱动配置环境

# yum install -y yum-utils device-mapper-persistent-data lvm2

4. 添加 docker yum repo

# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

5. 安装最新版本 docker-ce

# yum install docker-ce

二、安装 docker-compose

1. 安装epel源

# yum install -y epel-release

2. 安装docker-compose

1) 通过 yum 安装

# yum install -y docker-compose

2) 通过 pip 源安装

# pip install docker-compose

以上步骤安装的 docker-compose 可能非最新版本，导致执行编排任务提示升级版本

3) 通过下载压缩包安装

Docker-compose 与 Docker Engine 版本之间有以下约定，最好使用 pip 或者 curl 从官方下载安装：

Compose file format	Docker Engine
1	1.9.0+
2.0	1.10.0+
2.1	1.12.0+
2.2, 3.0, 3.1, 3.2	1.13.0+
2.3, 3.3, 3.4, 3.5	17.06.0+
2.4	17.12.0+
3.6	18.02.0+
3.7	18.06.0+

下载并安装最新版本1.26.2:

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

三、部署 es 集群

1. 初始化es 集群目录与配置

1) 创建映射目录

$ mkdir /data/elasticsearch && cd /data/elasticsearch
$ mkdir -pv es01/data es02/data es03/data es01/logs es02/logs es03/logs
$ chmod -R 777 es*

不修改权限，第一次启动的时候会提示错误信息：
Error opening log file 'logs/gc.log': Permission denied

2) 配置 elasticsearch

$ vi elasticsearch.conf

# network.host 设置允许其他ip访问，解除ip绑定
network.host: 0.0.0.0
# xpack.security 则是安全相关配置，其中ssl的证书需要自己生成
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.type: PKCS12

xpack.security.audit.enabled: true

3) 配置 kibana

$ vi kibana.conf

server.name: kibana
server.host: "0"
# elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: kibana
elasticsearch.password: ******

2. 创建 docker-compose.yml

$ vim /data/elasticsearch/docker-compose.yml

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es01/data:/usr/share/elasticsearch/data
      - ./es01/logs:/usr/share/elasticsearch/logs
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - 9200:9200
    networks:
      - elastic

   es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es02/data:/usr/share/elasticsearch/data
      - ./es02/logs:/usr/share/elasticsearch/logs
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - 9201:9200
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es03/data:/usr/share/elasticsearch/data
      - ./es03/logs:/usr/share/elasticsearch/logs
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - 9202:9200
    networks:
      - elastic

  kib01:
    depends_on:
      - es01
    image: docker.elastic.co/kibana/kibana:7.8.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elastic

networks:
  elastic:
    driver: bridge

3. 修改宿主机系统参数

$ sudo vim /etc/sysctl.conf

vm.max_map_count=262144

max_map_count文件包含限制一个进程可以拥有的VMA(虚拟内存区域)的数量。虚拟内存区域是一个连续的虚拟地址空间区域。在进程的生命周期中，每当程序尝试在内存中映射文件，链接到共享内存段，或者分配堆空间的时候，这些区域将被创建。调优这个值将限制进程可拥有VMA的数量。限制一个进程拥有VMA的总数可能导致应用程序出错，因为当进程达到了VMA上线但又只能释放少量的内存给其他的内核进程使用时，操作系统会抛出内存不足的错误。如果你的操作系统在NORMAL区域仅占用少量的内存，那么调低这个值可以帮助释放内存给内核用。
一般调整为默认的4倍。
官方解释：

“This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.

While most applications need less than a thousand maps, certain programs, particularly malloc debuggers, may consume lots of them, e.g., up to one or two maps per allocation.

The default value is 65536.”

立刻生效：

$ sysctl -w vm.max_map_count=262144

4. es 启用 SSL证书

配置证书 elastic-certificates.p12：

es提供了生成证书的工具elasticsearch-certutil，我们可以在docker实例中生成它，然后复制出来，后面统一使用。

首先运行 docker 实例

$ sudo docker run -dit --name=es elasticsearch:7.8.0 /bin/bash

生成ca: elastic-stack-ca.p12

[root@25dee1848942 elasticsearch]# ./bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 :

再生成cert: elastic-certificates.p12

[root@25dee1848942 elasticsearch]# ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.

这个生成elastic-certificates.p12 就是我们需要使用的，复制出证书, ctrl+d退出容器内部。

$ sudo docker cp es:/usr/share/elasticsearch/elastic-certificates.p12 .
# 关闭这个容器
$ sudo docker kill es
$ sudo docker rm es

5. 运行 docker-compose 创建容器，并生成密码

1) 创建es 容器集群

$ docker-compose up -d

2) 进入已经启动一台生成密码

$ sudo docker exec -it es01 /bin/bash

生成密码可以是可以使用 auto（自动生成密码），也可以自己设置密码：interactive 交互方式设置密码

[root@cfeeab4bb0eb elasticsearch]# ./bin/elasticsearch-setup-passwords -h
Sets the passwords for reserved users

Commands
--------
auto - Uses randomly generated passwords
interactive - Uses passwords entered by a user

Non-option arguments:
command              

Option             Description        
------             -----------        
-E <KeyValuePair>  Configure a setting
-h, --help         Show help          
-s, --silent       Show minimal output
-v, --verbose      Show verbose output



[root@cfeeab4bb0eb elasticsearch]# ./bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y


Changed password for user apm_system
PASSWORD apm_system = YxVzeT9B2jEDUjYp66Ws

Changed password for user kibana
PASSWORD kibana = 8NnThbj0N02iDaTGhidU

Changed password for user logstash_system
PASSWORD logstash_system = 9nIDGe7KSV8SQidSk8Dj

Changed password for user beats_system
PASSWORD beats_system = qeuVaf1VEALpJHfEUOjJ

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = DtZCrCkVTZsinRn3tW3D

Changed password for user elastic
PASSWORD elastic = q5f2qNfUJQyvZPIz57MZ

日志中可能的错误，从网上没有找到解决方案，google 中大多反馈不用理会：
es03     | "Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
es01     | "stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)"
参考：https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/#java-error-during-startup

6. 验证 es 服务

$ curl http://elastic:******@elasticsearch:9200

四、中文分词插件 elasticsearch-analysis-ik

1. 独立安装 analysis-ik

分别在三个节点：

$ cd /usr/share/elasticsearch/
$ ./bin/elasticsearch-plugin  install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.8.0/elasticsearch-analysis-ik-7.8.0.zip
$ cd plugins/analysis-ik/config
$ vim IKAnalyzer.cfg.xml

IKAnalyzer.cfg.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
        <comment>IK Analyzer 扩展配置</comment>
        <!--用户可以在这里配置自己的扩展字典 -->
        <entry key="ext_dict">./extra_single_word_full.dic</entry>
         <!--用户可以在这里配置自己的扩展停止词字典-->
        <entry key="ext_stopwords"></entry>
        <!--用户可以在这里配置远程扩展字典 -->
        <!-- <entry key="remote_ext_dict">words_location</entry> -->
        <!--用户可以在这里配置远程扩展停止词字典-->
        <!-- <entry key="remote_ext_stopwords">words_location</entry> -->
</properties>

也可以通过使用集成了 analysis-ik中文分词的Elasticsearch 镜像
$ docker run --rm --name es -p 9200:9200 -p 9300:9300 -v $(pwd)/data:/usr/share/elasticsearch/data windedge/elasticsearch-analysis-ik

2. 创建index 前去掉 HTML 标签

找一台机器，执行如下命令，创建过滤 HTML 标签 pipeline（若es 启用密码，需要输入）

curl --request PUT \
  -u elastic:****** \
  --url http://es-host:9200/_ingest/pipeline/escape_html \
  --header 'content-type: application/json' \
  --data '{
  "description": "清理html标签",
  "processors": [
    {
      "html_strip": {
        "field": "published_content",
        "if": "ctx.published_content != null"
      }
    },
    {
      "gsub": {
        "field": "published_content",
        "pattern": "\\n",
        "replacement": "",
        "if": "ctx.published_content != null"
      }
    }
  ]
}'

3. 重启 es 集群

$ docker-compose restart

4. 测试验证

$ curl --header 'content-type: application/json' --User elastic:****** 'http://localhost:9200/_analyze?pretty=true' -d '{"analyzer":"ik_smart","text":"我们是阿里巴巴国际旅行社（北京）有限公司"}'

{
  "tokens" : [
    {
      "token" : "我们",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "是",
      "start_offset" : 2,
      "end_offset" : 3,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "阿里",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "CN_WORD",
      "position" : 2
    },
    {
      "token" : "br",
      "start_offset" : 6,
      "end_offset" : 8,
      "type" : "ENGLISH",
      "position" : 3
    },
    {
      "token" : "巴巴",
      "start_offset" : 10,
      "end_offset" : 12,
      "type" : "CN_WORD",
      "position" : 4
    },
    {
      "token" : "国际",
      "start_offset" : 12,
      "end_offset" : 14,
      "type" : "CN_WORD",
      "position" : 5
    },
    {
      "token" : "旅行社",
      "start_offset" : 14,
      "end_offset" : 17,
      "type" : "CN_WORD",
      "position" : 6
    },
    {
      "token" : "北京",
      "start_offset" : 18,
      "end_offset" : 20,
      "type" : "CN_WORD",
      "position" : 7
    },
    {
      "token" : "有限公司",
      "start_offset" : 21,
      "end_offset" : 25,
      "type" : "CN_WORD",
      "position" : 8
    }
  ]
}

扩展知识

docker 镜像导出

# 导出镜像
$ docker save > image.tar image:1.0
# 导入镜像
$ docker load < image.tar
# 从容器中导出镜像
$ docker export -o nginx-test.tar nginx-test
# 将容器导出镜像导入
$ docker import nginx-test.tar nginx:imp

说明：export命令是从容器（container）中导出tar文件，而save命令则是从镜像（images）中导出

基于第二点，export导出的文件再import回去时，无法保留镜像所有历史（即每一层layer信息，不熟悉的可以去看Dockerfile），不能进行回滚操作；而save是依据镜像来的，所以导入时可以完整保留下每一层layer信息。如下图所示，nginx:latest是save导出load导入的，nginx:imp是export导出import导入的。

可以依据具体使用场景来选择命令:

若是只想备份images，使用save、load即可
若是在启动容器后，容器内容有变化，需要备份，则使用export、import

参考文章：Docker 镜像导入导出 https://blog.csdn.net/ncdx111/article/details/79878098

参考资料

es集群（包含中文分词插件）部署 https://www.cnblogs.com/tyhj-zxp/p/13166515.html
es集群部署 https://www.cnblogs.com/woshimrf/p/docker-es7.html

Doraman.Liu

关注

0
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
Elasticsearch 7.8 Docker 集群搭建

Elasticsearch 7.8 集群搭建Elasticsearch 7.8 集群搭建一、安装 docker 环境1. 更新 centos2. 卸载旧版本 docker3. 安装docker的驱动配置环境4. 添加 docker yum repo5. 安装最新版本 docker-ce二、安装 docker-compose1. 安装epel源2. 安装docker-compose1) 通过 yum 安装2) 通过 pip 源安装3) 通过下载压缩包安装三、部署 es 集群1. 初始化es 集群目录与配置1)
复制链接

扫一扫