Docker方式搭建ELKF集群--集成kafka和redis抽取日志

参考网站：Docker方式搭建ELK集群_ubuntu docker 搭建elk 集群-CSDN博客

节点名称	Ip	角色
es01	192.168.88.133	主节点
es02	192.168.88.139	从节点1
es03	192.168.88.140	从节点2

一、安装部署 Elasticsearch 版本：7.11.2

1.每个节点配置docker加速

编辑文件

sudo vim /etc/docker/daemon.conf

添加如下

{



"registry-mirrors": [

     "https://docker.mirrors.ustc.edu.cn",

     "https://registry.docker-cn.com",

     "http://hub-mirror.c.163.com",

     "https://mirror.ccs.tencentyun.com",

     "http://f1361db2.m.daocloud.io",

     "https://cr.console.aliyun.com",

     "https://hub.docker.com",

     "https://hub.daocloud.io/",

     "https://c.163.com/hub#/m/home/",

     "http://ef017c13.m.daocloud.io"],

    "insecure-registries": ["10.0.0.12:5000"]



}

从启动docker服务

sudo systemctl daemon-reload

sudo systemctl restart docker

如果重启报错：Job for docker.service failed because the control process exited with error code.

See "systemctl status docker.service" and "journalctl -xe" for details.

参考目录“一点问题解决”

2、每个节点建立docker网络

sudo docker network create -d bridge elastic

3、每个节点拉取docker镜像

sudo docker pull elasticsearch:7.11.2

三个节点分别创建文件夹和配置文件

192.168.88.133 上创建目录

/home/es01/config   

/home/es01/data

请确保 /home/es01/data 目录有写的权限！
可以先直接 chmod -R 777 /home/es01/ 授权

在主节点上创建文件 home/es01/config/elasticsearch.yml 

cluster.name: es-docker-cluster

node.name: es01

node.master: true

node.data: true

network.host: 0.0.0.0

http.port: 9200

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.88.133","192.168.88.139","192.168.88.140"]

discovery.zen.minimum_master_nodes: 3

gateway.recover_after_nodes: 2

cluster.initial_master_nodes: ["es01","es02","es03"]

http.cors.enabled: true

http.cors.allow-origin: "*"

network.publish_host: 192.168.88.133

192.168.88.139 上创建目录

/home/es02/config

/home/es02/data

请确保 /home/es02/data 目录有写的权限！
可以先直接 chmod -R 777 /home/es02/ 授权

在从节点1上创建文件 home/es01/config/elasticsearch.yml

cluster.name: es-docker-cluster

node.name: es02

node.master: true

node.data: true

network.host: 0.0.0.0

http.port: 9200

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.88.133","192.168.88.139","192.168.88.140"]

discovery.zen.minimum_master_nodes: 3

gateway.recover_after_nodes: 2

cluster.initial_master_nodes: ["es01","es02","es03"]

http.cors.enabled: true

http.cors.allow-origin: "*"

network.publish_host: 192.168.88.139

192.168.88.140 上创建目录

/home/es03/config

/home/es03/data

请确保 /home/es03/data 目录有写的权限！
可以先直接 chmod -R 777 /home/es031/ 授权

在从节点2上创建文件 home/es01/config/elasticsearch.yml

cluster.name: es-docker-cluster

node.name: es03

node.master: true

node.data: true

network.host: 0.0.0.0

http.port: 9200

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.88.133","192.168.88.139","192.168.88.140"]

discovery.zen.minimum_master_nodes: 3

gateway.recover_after_nodes: 2

cluster.initial_master_nodes: ["es01","es02","es03"]

http.cors.enabled: true

http.cors.allow-origin: "*"

network.publish_host: 192.168.88.140

各个节点内核参数文件 /etc/sysctl.conf 里面增加一行 vm.max_map_count=262144

net.ipv4.ip.forward=1

vim  /etc/sysctl.conf

sysctl -p #使其生效

上面 必须注意： 如果要生效 /etc/sysctl.conf 请务必确保 net.ipv4.ip_forward=1 ，默认他是0，如果是0

则ip4转发功能将限制， 很可能本机上 docker版本的nginx 全部抛出502错误 ，启动docker端口映射则会抛出“ IPv4 forwarding is disabled. Networking will not work” 错误！

4、主节点执行docker脚本

docker run --restart=always  --network=elastic \

-d -p 9200:9200 -p 9300:9300 \

-e ES_JAVA_OPTS="-Xms4g -Xmx4g" \

-e "discovery.type=zen" \

-v /home/es01/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /home/es01/data:/usr/share/elasticsearch/data \

--name es01 elasticsearch:7.11.2

5、从节点1执行docker脚本

docker run --restart=always  --network=elastic \

-d -p 9200:9200 -p 9300:9300 \

-e ES_JAVA_OPTS="-Xms4g -Xmx4g" \

-e "discovery.type=zen" \

-v /home/es02/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /home/es02/data:/usr/share/elasticsearch/data \

--name es02 elasticsearch:7.11.2

6、从节点2执行docker脚本

docker run --restart=always  --network=elastic \

-d -p 9200:9200 -p 9300:9300 \

-e ES_JAVA_OPTS="-Xms4g -Xmx4g" \

-e "discovery.type=zen" \

-v /home/es03/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \

-v /home/es03/data:/usr/share/elasticsearch/data \

--name es03 elasticsearch:7.11.2

各个节点执行curl 192.168.88.133:9200/_cat/nodes可以看到

1. 设置登录密码

进入主节点容器

docker exec -it es01 /bin/bash

生成集群证书（集群的证书三个节点必须一样）