Docker-compose一键部署3台(3=N)Elasticsearch+kibana集群(kibana可摘离)
版本号(docker相关下载以及安装,请看以前文章。)
- Docker version 20.10.17, build 100c701(yum安装)
- docker-compose-linux-x86_64(离线安装)
- Elasticsearch:7.14.2
- Kibana:7.14.2
1、创建集群目录
for i in `seq 1 3`
do
mkdir -p /home/data/es/config
mkdir -p /home/data/es/data-$i
mkdir -p /home/data/es/log-$i
mkdir -p /home/data/es/plugins-$i
mkdir -p /home/data/kibana/config
done
2、授权es权限
chmod 777 /home/data/es
3、配置文件
1)Elasticsearch配置文件(初始)
#集群名称
cluster.name: my-es
#节点将绑定到一个主机名或者 ip 地址并且会将该这个节点通知集群中的其他节点。接受 ip 地址,主机名,指定值或者包含这些值的数组,默认值:local
network.host: 0.0.0.0
#表示是否可以充当主节点
node.master: true
#表示是否允当数据节点
node.data: true
#是否支持跨域,默认为false
http.cors.enabled: true
#表示开启xpack认证机制,使用密码登入
#xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
#xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
2)kibana配置文件(初始)
# ** THIS IS AN AUTO-GENERATED FILE **
#
# Default Kibana configuration for docker target
server.shutdownTimeout: "5s"
#Elasticsearch集群地址
elasticsearch.hosts: [ "http://172.18.0.2:9200","http://172.18.0.3:9201","http://172.18.0.4:9202" ]
#网关地址
server.host: "0.0.0.0"
#Kibana映射端口
server.port: 5601
#elasticsearch.username: "kibana"
#elasticsearch.password: "mola2022"
monitoring.ui.container.elasticsearch.enabled: true
# 支持中文
i18n.locale: "zh-CN"
3)docker-compose.yaml(初始,格式一定要正确)
version: '3'
services:
es01:
image: elasticsearch:7.14.2
container_name: es01
environment:
- node.name=es01
- cluster.name=my-es
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
# volumes:
# - /home/data/es/plugins-1:/usr/share/elasticsearch/plugins
# - /home/data/es/data-1:/usr/share/elasticsearch/data
# - /home/data/es/config/es01.yml:/usr/share/elasticsearch/config/elasticsearch.yml
# - /home/data/es/log-1:/usr/share/elasticsearch/logs
# - /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
# ports:
# - "9200:9200"
# - "9300:9300"
# networks:
# es:
# ipv4_address: 172.18.0.2
es02:
image: elasticsearch:7.14.2
container_name: es02
environment:
- node.name=es02
- cluster.name=my-es
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
# volumes:
# - /home/data/es/plugins-2:/usr/share/elasticsearch/plugins
# - /home/data/es/data-2:/usr/share/elasticsearch/data
# - /home/data/es/config/es02.yml:/usr/share/elasticsearch/config/elasticsearch.yml
# - /home/data/es/log-2:/usr/share/elasticsearch/logs
# - /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
# ports:
# - "9201:9200"
# - "9301:9300"
# networks:
# es:
# ipv4_address: 172.18.0.3
es03:
image: elasticsearch:7.14.2
container_name: es03
environment:
- node.name=es03
- cluster.name=my-es
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
# volumes:
# - /home/data/es/plugins-3:/usr/share/elasticsearch/plugins
# - /home/data/es/data-3:/usr/share/elasticsearch/data
# - /home/data/es/config/es03.yml:/usr/share/elasticsearch/config/elasticsearch.yml
# - /home/data/es/log-3:/usr/share/elasticsearch/logs
# - /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
# ports:
# - "9202:9200"
# - "9302:9302"
# networks:
# es:
# ipv4_address: 172.18.0.4
# 根据自己需求添加kibana
# kibana:
# image: kibana:7.14.2
# container_name: kibana
# restart: always
# ports:
# - 5601:5601
# volumes:
# - /home/data/kibana/config:/opt/kibana/config
# networks:
# es:
# ipv4_address: 172.18.0.5
#networks:
# es:
# driver: bridge
# ipam:
# config:
# - subnet: "172.18.0.0/24"
4)目录结构
[root@localhost data]# tree ./
./
├── es
│ ├── config
│ │ ├── docker-compose.yaml
│ │ ├── es01.yml
│ │ ├── es02.yml
│ │ └── es03.yml
│ ├── data-1
│ ├── data-2
│ ├── data-3
│ ├── log-1
│ ├── log-2
│ ├── log-3
│ ├── plugins-1
│ ├── plugins-2
│ └── plugins-3
└── kibana
└── config
└── kibana.yml
4、启动集群
1)启动集群,设置认证证书(防止报错,一路回车就好)
docker-compose up -d
docker exec -it es01 /bin/bash
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
2)把容器中的证书备份出来,并且给与权限,备用
cd /home/data/es/config
docker cp es01:/usr/share/elasticsearch/elastic-certificates.p12 ./
chmod 777 elastic-certificates.p12
3)关闭容器
docker-compose down
4)打开docker-compose.yaml所有es、network注释的配置文件
version: '3'
services:
es01:
image: elasticsearch:7.14.2
container_name: es01
environment:
- node.name=es01
- cluster.name=my-es
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /home/data/es/plugins-1:/usr/share/elasticsearch/plugins
- /home/data/es/data-1:/usr/share/elasticsearch/data
- /home/data/es/config/es01.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/data/es/log-1:/usr/share/elasticsearch/logs
- /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
ports:
- "9200:9200"
- "9300:9300"
networks:
es:
ipv4_address: 172.18.0.2
es02:
image: elasticsearch:7.14.2
container_name: es02
environment:
- node.name=es02
- cluster.name=my-es
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /home/data/es/plugins-2:/usr/share/elasticsearch/plugins
- /home/data/es/data-2:/usr/share/elasticsearch/data
- /home/data/es/config/es02.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/data/es/log-2:/usr/share/elasticsearch/logs
- /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
ports:
- "9201:9200"
- "9301:9300"
networks:
es:
ipv4_address: 172.18.0.3
es03:
image: elasticsearch:7.14.2
container_name: es03
environment:
- node.name=es03
- cluster.name=my-es
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
#根据自己需求更改
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /home/data/es/plugins-3:/usr/share/elasticsearch/plugins
- /home/data/es/data-3:/usr/share/elasticsearch/data
- /home/data/es/config/es03.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/data/es/log-3:/usr/share/elasticsearch/logs
- /home/data/es/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
ports:
- "9202:9200"
- "9302:9302"
networks:
es:
ipv4_address: 172.18.0.4
# kibana:
# image: kibana:7.14.2
# container_name: kibana
# restart: always
# ports:
# - 5601:5601
# volumes:
# - /home/data/kibana/config:/opt/kibana/config
# networks:
# es:
# ipv4_address: 172.18.0.5
networks:
es:
driver: bridge
ipam:
config:
- subnet: "172.18.0.0/24"
5)进入容器设置密码(手动)
docker exec -it es01 /bin/bash
./bin/elasticsearch-setup-passwords interactive
6)登入网页验证
账号:elastic 密码:你设置的密码
5、加入Kibana
1)打开Kibana配置文件注释
vim /home/data/kibana/config/kibana.yml
# ** THIS IS AN AUTO-GENERATED FILE **
#
# Default Kibana configuration for docker target
server.shutdownTimeout: "5s"
#Elasticsearch集群地址
elasticsearch.hosts: [ "http://172.18.0.2:9200","http://172.18.0.3:9201","http://172.18.0.4:9202" ]
#网关地址
server.host: "0.0.0.0"
#Kibana映射端口
server.port: 5601
elasticsearch.username: "kibana"
elasticsearch.password: "mola2022"
monitoring.ui.container.elasticsearch.enabled: true
# 支持中文
i18n.locale: "zh-CN"
2)方法1:(分离)docker run
docker run -id -v /home/data/kibana/config:/opt/kibana/config -p 5601:5601 -m 512m --restart=always --memory-swap=1024m --name kibana --network config_es kibana:7.14.2
3)方法2:打开docker-compose关于kibana注释相关信息
kibana:
image: kibana:7.14.2
container_name: kibana
restart: always
ports:
- 5601:5601
volumes:
- /home/data/kibana/config:/opt/kibana/config
networks:
es:
ipv4_address: 172.18.0.5
6、本文涉及的其他命令、知识
1)命令
#查看docker网络
docker network ls
#删除docker网络config_es
docker network rm config_es
#查看es01容器网络信息
docker inspect es01 |grep -A 10 'IPAddress'
#启用另一个docker-compose.yaml文件
docker-compose -f 11.yaml up -d
2)Elasticsearch其他相关配置文件(可以根据需求增加,但是用docker-compose有的会冲突,注意看报错)
#集群名称
cluster.name: cluster-es
#节点名称,每个节点的名称不能重复
node.name: node-1
#ip地址,每个节点的地址不能重复
network.host: linux1 #或者ip
#是不是有资格主节点
node.master: true
#表示是否允当数据节点
node.data: true
#http连接端口
http.port: 9200
#tcp传输端口
transport.tcp.port: 9300
#head 插件需要这打开两个配置
http.cors.allow-origin: "*"
#是否支持跨域,默认为false
http.cors.enabled: true
#节点将绑定到一个主机名或者 ip 地址并且会将该这个节点通知集群中的其他节点。接受 ip 地址,主机名,指定值或者包含这些值的数组,默认值:local
network.host: 0.0.0.0
#http最大请求连接量
http.max_content_length: 200mb
#es7.x之后新增的配置,初始化一个新的集群时需要此配置来选举master
cluster.initial_master_nodes: ["node-1"]
#es7.x之后新增的配置,节点发现
discovery.seed_hosts: ["linux1:9300","linux2:9300","linux3:9300"]
#为了加入集群,一个节点至少需要知道集群中其他节点的主机名或者 ip 地址。这个设置提供初始其他节点列表,当前节点将尝试联系。接收 ip 地址或者主机名。默认值:["127.0.0.1", "[::1]"]
discovery.zen.ping.unicast.hosts
#预期在集群中的(数据或主)节点数。只要预期的节点数已加入集群,就会启动本地分片的恢复。默认为0
gateway.expected_nodes
#预期在集群中的主节点数。一旦预期的主节点数加入集群,就会开始恢复本地分片。默认为0
gateway.expected_master_nodes
#预期在集群中的数据节点数。一旦预期数量的节点已加入集群,就会启动本地分片的恢复。默认为0
gateway.expected_data_nodes
#如果未达到预期的节点数,则恢复过程将等待配置的时间量,然后再尝试恢复。如果只要配置了expected_nodes,则默认这个参数值为5m,一旦recover_after_time持续时间超时,只要满足以下条件,恢复就会开始:
gateway.recover_after_time
#只要此许多数据或主节点已加入集群,即可恢复。
gateway.recover_after_nodes
#只要这么多主节点已加入集群,就可以恢复。
gateway.recover_after_master_nodes: 2
#只要这么多数据节点已加入集群,就可以恢复。
gateway.recover_after_data_nodes
#开启或关闭 TCP 长连接,默认值为 true。
network.tcp.keep_alive: true
#开启或关闭 TCP 无延迟设置。默认值为 true。
network.tcp.no_delay: true
#一个地址是否可以被重用。在非 windows 机子上默认值为 true。
network.tcp.reuse_address
#TCP 发送缓冲区大小(以size unit指定)。没有默认值。
network.tcp.send_buffer_size
#TCP 接收缓冲区大小(以size unit指定)。没有默认值。
network.tcp.receive_buffer_size
transport.tcp.compress: true
#集群内同时启动的数据任务个数。默认是2个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或者删除节点及负载均衡时并发恢复的线程个数,默认4个
cluster.routing.allocation.node_concurrent_recoveriesL: 16
#初始化数据恢复时,并发恢复线程的个数,默认为4个
cluster.routing.allocation.node_initial_primaries_recoveries: 16
#表示开启xpack认证机制,使用密码登入
xpack.security.enabled: true
#这条如果不配,es将起不来,会报如下错误,Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
xpack.security.transport.ssl.enabled: true
#认证证书类型,目录位置
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
找了很多资料,很多都是相同的,所以想自己写一篇,亲测可用。
对大家有用的话,可以点个关注~~你们的支持,是我最大的动力。