docker-compose部署6.8.23版本elasticsearch+es-head+kibana多节点集群及部分排错处理

序言

现阶段很多elasticsearch部署都是7版本之上的，但部分要求是6.8版本，俩者很多配置参数存在差异问题，elasticsearch有状态服务，kubernetes和docker启动有点麻烦，最好用docker-compose启动，可以保证数据稳定。

主机信息

节点信息	hosts信息
192.168.121.137	es-master
192.168.121.138	es-node1

前提工作

本机hosts解析

master执行，node改下主机名

#hostnamectl set-hostname es-master

#cat /etc/hosts
192.168.121.137 es-master
192.168.121.138 es-node1

增加配置

# cat /etc/sysctl.conf
vm.max_map_count=655360
# sysctl -p

自建的虚拟机

需要关闭selinux 设置为disabled，docker容器启动后关闭防火墙，切勿提前关闭，可能会导致报错，若发生报错，重启docker即可。
需要配置centos的源和epel源，如下链接可以下载。

https://developer.aliyun.com/mirror/

#cat /etc/sysconfig/selinux
SELINUX=disabled

云服务器

检查selinux是否关闭，防火墙根据安全组进行管理

#cat /etc/sysconfig/selinux
SELINUX=disabled

操作

本次操作系统为centos7，docker版本为20.10.12，docker-compose版本为 v2.12.2，可以自由波动，如下是对应关系。

docker安装

俩节点一样操作
离线安装包地址:

https://download.docker.com/linux/static/stable/x86_64/

用国内源加载

# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

卸载旧docker

#yum remove -y docker \
              docker-client \
              docker-client-latest \
              docker-common \
              docker-latest \
              docker-latest-logrotate \
              docker-logrotate \
              docker-selinux \
              docker-engine-selinux \
              docker-engine \
              container*

查看源包含的docker版本，风格自由

#yum list docker-ce --showduplicates | sort -r

安装docker指定版本

#yum -y install docker-ce-20.10.12-3.el7 docker-ce-cli-20.10.12-3.el7

设置docker镜像加速，地址可以去阿里云acr产品控制台获取

#vi /etc/docker/daemon.json

{
  "registry-mirrors": ["https://xxxxxxxxx.mirror.aliyuncs.com"]
}

启动docker，开机自启

#systemctl start docker
#systemctl enable docker
#systemctl status docker

安装docker-compose

俩节点一样操作
离线安装包地址：

https://github.com/docker/compose/releases/download/v2.12.2/docker-compose-linux-x86_64

国内下载地址

#curl -Lk https://get.daocloud.io/docker/compose/releases/download/v2.12.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

在线下载

#curl -SL https://github.com/docker/compose/releases/download/v2.12.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

添加可执行权限

#chmod +x /usr/local/bin/docker-compose

版本测试

#docker-compose --version
输出：Docker Compose version v2.12.2

部署es集群

注意：有规定就创建普通用户，没要求的就用root启动，本次操作用的root，普通用户的话后面的目录自我授权即可

es-master操作

1、创建部署目录

#mkdir /root/es && cd /root/es

2、编写docker-compose.yml文件
JAVA_OPTS可以根据节点资源进行设置，建议先pull下elasticsearch:6.8.23、elasticsearch-head:6-alpine镜像到本地

version: '3'
services:
  es-master:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.23
    container_name: es-master
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - /data/elasticsearch/config/es.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - /data/elasticsearch/data:/usr/share/elasticsearch/data:rw
      - /data/elasticsearch/log:/usr/share/elasticsearch/log:rw
    ports:
      - 9200:9200
      - 9300:9300
    extra_hosts:
      - "es-master:192.168.121.137"
      - "es-node1:192.138.121.138"
  elasticsearch-head:
      image: wallbase/elasticsearch-head:6-alpine
      container_name: elasticsearch-head
      environment:
        TZ: 'Asia/Shanghai'
      ports:
        - '9100:9100'

3、创建es配置数据日志目录和授权

mkdir /data/elasticsearch/{config,data,log} -pv

chmod 777 -R /data/elasticsearch

4、编辑es.yml文件

cd /data/elasticsearch/config

#cat es.yml
cluster.name: elasticsearch-cluster
node.name: es-master
network.bind_host: 0.0.0.0
network.publish_host: 192.168.121.137
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,Content-Type
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["es-master:9300","es-node1:9300"]
discovery.zen.minimum_master_nodes: 1

es-node1操作

1、创建部署目录

#mkdir /root/es && cd /root/es

2、编写docker-compose.yml文件
JAVA_OPTS可以根据节点资源进行设置，建议先pull下elasticsearch:6.8.23到本地

version: '3'
services:
  es-node1:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.23
    container_name: es-node1
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - /data/elasticsearch/config/es.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - /data/elasticsearch/data:/usr/share/elasticsearch/data:rw
      - /data/elasticsearch/log:/usr/share/elasticsearch/log:rw
    ports:
      - 9200:9200
      - 9300:9300
    extra_hosts:
      - "es-master:192.168.121.137"
      - "es-node1:192.138.121.138"

3、创建es配置数据日志目录和授权

mkdir /data/elasticsearch/{config,data,log} -pv

chmod 777 -R /data/elasticsearch

4、编辑es.yml文件

cd /data/elasticsearch/config

#cat es.yml
cluster.name: elasticsearch-cluster
node.name: es-node1
network.bind_host: 0.0.0.0
network.publish_host: 192.168.121.138
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,Content-Type
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["es-master:9300","es-node1:9300"]
discovery.zen.minimum_master_nodes: 1

es-master、es-node1 docker-compose启动

#pwd
/root/es
#docker-compose up -d

启动es_xpack 认证

es-master操作
1、登录其中一个节点的容器内，生成证书

#docker exec -it es-master bash
# /usr/share/elasticsearch/bin/elasticsearch-certutil ca
# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
## 两条命令均一路回车即可，不需要给秘钥再添加密码。

拷贝到宿主机挂载目录

# mv elastic-* /usr/share/elasticsearch/data/

## 复制 /data/elasticsearch/data/ 下证书到 config 目录
# cd /data/elasticsearch/config/
# cp /data/elasticsearch/data/elastic-*  ./
# chmod 644 elastic-*

复制证书文件到其他节点

# scp /data/elasticsearch/config/elastic-* es-node1:/data/elasticsearch/config/

es-master、es-node1一致操作
新增es.yml配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.audit.enabled: true

修改docker-compose.yml文件
volumes下追加一条

- /data/elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12:ro

尝试启动可能会报错
1、Caused by: java.lang.IllegalArgumentException: unknown setting [xck.security.enabled] did you mean any of [xpack.security.enabled, xpack.security.audit.enabled]?
解决：
在es.yml文件内去掉这部分setting

2、Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/elastic-certificates.p12
解决：
检查文件是否在，若在，查看下权限，root账号启动的需要将文件同步到容器内，授予权限

3、Caused by: java.security.AccessControlException: access denied (“java.io.FilePermission” “/data/elasticsearch/config/elastic-certificates.p12” “read”)
如上排查一致

设置密码
ES 中内置了几个管理其他集成组件的账号即：apm_system, beats_system, elastic, kibana, logstash_system, remote_monitoring_user，使用之前，首先需要添加一下密码。
我设置的密码跟账号一致

# docker exec -it es-mater bash
# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

访问es-head

curl http://192.168.121.137:9100/?auth_user=elastic&auth_password=elastic

可能出现的问题是集群不健康，显示未连接
1、可能自建虚拟机的需要停掉容器，重启虚拟机，云服务器的切勿操作，生产耦合业务机器切勿操作
2、经过了加密的都需要在es.yml内增加一个参数

http.cors.allow-headers: Authorization,Content-Type

重新启动
3、访问链接检查是否有误，密码是否有误
4、连接地址最好填准确的ip地址，不用默认的localhost

加密失败

原因：Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
1、可能路劲不对，2、删除es data目录下的数据再启动

xpack离线安装

安装教程

部署kibana

1、es-master操作

#docker pull kibana:6.8.13

docker-compose.yml文件追加一条service，注意对齐

 kibana:
      image: kibana:6.8.13
      restart: always
      container_name: kibana
      volumes:
        - /data/elasticsearch/config/kibana.yml:/usr/share/kibana/config/kibana.yml
      ports:
        - '5601:5601'     #java、集群通信端口
      privileged: true    #环境变量

配置kibana.yml

#cat /data/elasticsearch/config/kibana.yml
server.name: kibana
# kibana的主机地址 0.0.0.0可表示监听所有IP
server.host: "0.0.0.0"
# # kibana访问es的URL
elasticsearch.hosts: [ "http://192.168.121.137:9200","http://192.168.121.138:9200" ]
elasticsearch.username: 'elastic'   #xpack设置的es账号密码也可以用kibana设置的账密，看情况决定
elasticsearch.password: 'elastic'
xpack.monitoring.ui.container.elasticsearch.enabled: false #显示cpu监控指标

2、es-master和es-node1操作
es.yml追加该配置用于监控

xpack.monitoring.collection.enabled: true

重启服务，加载yml配置

#docker-compose restar
##或者
#docker-compose restart 指定service

访问kibana

http://192.168.121.137:5601/

账密

elastic/elastic

问题1、Caused by: java.lang.IllegalStateException: index and alias names need to be unique, but the following duplicates were found [.kibana (alias of [.kibana_1/OBxJe4x1SFauudkY64PgQw])]
解决：
删掉该索引目录，别担心集群化，你es起来后会生成一个一样的索引目录。