docker部署es集群

docker部署es集群

es的版本：7.8.0

拉取es镜像

docker pull elasticsearch:7.8.0

设置 max_map_count

• 这个是虚拟机层面的，所以只需要配置一次

echo 'vm.max_map_count=262144' >>/etc/sysctl.conf
sysctl -p

准备虚拟网络和挂载目录

# 创建虚拟网络
docker network create es-net

# node0 的挂载目录
mkdir -p -m 777 /var/lib/es/node0/config
mkdir -p -m 777 /var/lib/es/node0/plugins
mkdir -p -m 777 /var/lib/es/node0/data

# node0 的挂载目录
mkdir -p -m 777 /var/lib/es/node1/config
mkdir -p -m 777 /var/lib/es/node1/plugins
mkdir -p -m 777 /var/lib/es/node1/data

# node2 的挂载目录
mkdir -p -m 777 /var/lib/es/node2/config
mkdir -p -m 777 /var/lib/es/node2/plugins
mkdir -p -m 777 /var/lib/es/node2/data

# node3 的挂载目录
mkdir -p -m 777 /var/lib/es/node3/config
mkdir -p -m 777 /var/lib/es/node3/plugins
mkdir -p -m 777 /var/lib/es/node3/data

配置每个节点的es.yml文件

es9200

cluster.name: dockerEs-cluster
node.name: es9200
network.bind_host: 0.0.0.0
network.publish_host: 172.16.37.120
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["172.16.37.120:9300","172.16.37.120:9301","172.16.37.120:9302","172.16.37.120:9303"]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: ["es9200","es9201","es9202","es9203"]
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

es9201

cluster.name: dockerEs-cluster
node.name: es9201
network.bind_host: 0.0.0.0
network.publish_host: 172.16.37.120
http.port: 9201
transport.tcp.port: 9301
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["172.16.37.120:9300","172.16.37.120:9301","172.16.37.120:9302","172.16.37.120:9303"]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: ["es9200","es9201","es9202","es9203"]
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

es9202

cluster.name: dockerEs-cluster
node.name: es9202
network.bind_host: 0.0.0.0
network.publish_host: 172.16.37.120
http.port: 9202
transport.tcp.port: 9302
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["172.16.37.120:9300","172.16.37.120:9301","172.16.37.120:9302","172.16.37.120:9303"]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: ["es9200","es9201","es9202","es9203"]
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

es9203

cluster.name: dockerEs-cluster
node.name: es9203
network.bind_host: 0.0.0.0
network.publish_host: 172.16.37.120
http.port: 9203
transport.tcp.port: 9303
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["172.16.37.120:9300","172.16.37.120:9301","172.16.37.120:9302","172.16.37.120:9303"]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: ["es9200","es9201","es9202","es9203"]
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type                                       

启动 Elasticsearch 集群

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d --net es-net -p 9200:9200 -p 9300:9300 -v /var/lib/es/node0/config/es9200.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /var/lib/es/node0/data:/usr/share/elasticsearch/data --name ES9200 elasticsearch:7.8.0

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d --net es-net -p 9201:9201 -p 9301:9301 -v /var/lib/es/node1/config/es9201.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /var/lib/es/node1/data:/usr/share/elasticsearch/data --name ES9201 elasticsearch:7.8.0

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d --net es-net -p 9202:9202 -p 9302:9302 -v /var/lib/es/node2/config/es9202.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /var/lib/es/node2/data:/usr/share/elasticsearch/data --name ES9202 elasticsearch:7.8.0

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d --net es-net -p 9203:9203 -p 9303:9303 -v /var/lib/es/node3/config/es9203.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /var/lib/es/node3/data:/usr/share/elasticsearch/data --name ES9203 elasticsearch:7.8.0

es节点安装searchGuard插件

es对应的searchGuard版本：search-guard-suite-plugin-7.8.0-43.2.0

插件拷贝至容器内

$ docker cp search-guard-suite-plugin-7.8.0-43.2.0 es9200:/opt/

• 命令中es9200就是容器的id，可以通过docker ps -a 查看容器id

登录容器并安装

$ docker exec -it es9200 bash
$ /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///opt/search-guard-6-6.6.2-24.2.zip

将searchGuard安装完成后，就可以着手进行凭证生成了。凭证的生成和插件安装无关联，可以理解就是个类似ssl证书的东西，有了这个东西后面才能配置searchGuard证书位置

searchGuard证书生成

使用search-guard-tlstools进行证书生成

版本：search-guard-tlstool-1.9.1

安装解压

正常操作，可以用linux生成，也可以本地生成。

TLS生成证书配置

• 复制/config/example.yml 并修改成<tlstool-1.6>/config/tlsconfig.yml(名字随意)
• yibai 是公司名称
• ca: 根证书配置
• node: 节点证书配置
• clients: 客户端证书配置

###
### Self-generated certificate authority
###
#
# If you want to create a new certificate authority, you must specify its parameters here.
# You can skip this section if you only want to create CSRs
#
ca:
  root:
      # The distinguished name of this CA. You must specify a distinguished name.   
     dn: CN=root.ca.yibai.com,OU=CA,O=yibai Com\, Inc.,DC=yibai,DC=com
 
      # The size of the generated key in bits
     keysize: 2048
 
      # The validity of the generated certificate in days from now
     validityDays: 3650
     
      # Password for private key
      #   Possible values:
      #   - auto: automatically generated password, returned in config output; 
      #   - none: unencrypted private key; 
      #   - other values: other values are used directly as password   
     pkPassword: auto
     
      # The name of the generated files can be changed here
     file: root-ca.pem
     
   # If you want to use an intermediate certificate as signing certificate,
   # please specify its parameters here. This is optional. If you remove this section,
   # the root certificate will be used for signing.         
  intermediate:
      # The distinguished name of this CA. You must specify a distinguished name.
     dn: CN=signing.ca.yibai.com,OU=CA,O=yibai Com\, Inc.,DC=yibai,DC=com
  
      # The size of the generated key in bits   
     keysize: 2048
     
      # The validity of the generated certificate in days from now      
     validityDays: 3650
 
     pkPassword: auto
           
      # If you have a certificate revocation list, you can specify its distribution points here      
     crlDistributionPoints: URI:https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl
 
###
### Default values and global settings
###
defaults:
 
      # The validity of the generated certificate in days from now
     validityDays: 3650
     
      # Password for private key
      #   Possible values:
      #   - auto: automatically generated password, returned in config output; 
      #   - none: unencrypted private key; 
      #   - other values: other values are used directly as password   
     pkPassword: auto      
     
      # Specifies to recognize legitimate nodes by the distinguished names
      # of the certificates. This can be a list of DNs, which can contain wildcards.
      # Furthermore, it is possible to specify regular expressions by
      # enclosing the DN in //.
      # Specification of this is optional. The tool will always include
      # the DNs of the nodes specified in the nodes section.            
      #nodesDn:
      #- "CN=*.yibai.com,OU=Ops,O=yibai Com\\, Inc.,DC=yibai,DC=com"
      # - 'CN=node.other.com,OU=SSL,O=Test,L=Test,C=DE'
      # - 'CN=*.yibai.com,OU=SSL,O=Test,L=Test,C=DE'
      # - 'CN=elk-devcluster*'
      # - '/CN=.*regex/'
 
      # If you want to use OIDs to mark legitimate node certificates,
      # the OID can be included in the certificates by specifying the following
      # attribute
     
      # nodeOid: "1.2.3.4.5.5"
 
      # The length of auto generated passwords            
     generatedPasswordLength: 12
     
      # Set this to true in order to generate config and certificates for
      # the HTTP interface of nodes
     httpsEnabled: true
     
      # Set this to true in order to re-use the node transport certificates
      # for the HTTP interfaces. Only recognized if httpsEnabled is true
     
      # reuseTransportCertificatesForHttp: false
     
      # Set this to true to enable hostname verification
      #verifyHostnames: false
     
      # Set this to true to resolve hostnames
      #resolveHostnames: false
     
     
###
### Nodes
###
#
# Specify the nodes of your ES cluster here
#      
nodes:
 - name: node1
   dn: CN=node1.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
   dns: node1.yibai.com
   ip: 192.168.71.246
 - name: node2
   dn: CN=node2.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
   dns:
     - node2.yibai.com
     - es2.yibai.com
   ip:
     - 10.0.2.1
     - 192.168.2.1
 - name: node3
   dn: CN=node3.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
   dns: node3.yibai.com
 
###
### Clients
###
#
# Specify the clients that shall access your ES cluster with certificate authentication here
#
# At least one client must be an admin user (i.e., a super-user). Admin users can
# be specified with the attribute admin: true    
#        
clients:
 - name: spock
   dn: CN=spock.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
 - name: kirk
   dn: CN=kirk.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
   admin: true

生成证书

<tlstool directory>/tools/sgtlstool.sh -c ../config/tlsconfig.yml -ca -crt

复制证书到es

# 复制节点证书到es配置目录
cp <tlstool directory>/tools/out/<nodename>.pem <ES directory>/config/
cp <tlstool directory>/tools/out/<nodename>.key <ES directory>/config/
cp <tlstool directory>/tools/out/<nodename>_http.pem <ES directory>/config/
cp <tlstool directory>/tools/out/<nodename>_http.key <ES directory>/config/
# 复制根证书到es配置目录
cp <tlstool directory>/tools/out/root-ca.pem <ES directory>/config/
# 复制客户端证书到es配置目录
cp <tlstool directory>/tools/out/spock.pem <ES directory>/config/
cp <tlstool directory>/tools/out/spock.key <ES directory>/config/

注：上方命令是普通的复制，如果是docker里面的话，还是用docker cp es9200.pem c14146c6076d:/opt/,然后再进入到容器里面移动到es的config文件夹里面。

修改节点内elastic search.yml配置文件

# 集群名称
cluster.name: ebuy-cloud-cluster
 
# 节点名
node.name: node-1
 
# 数据存储路径
path.data: /data/es_data
 
# 日志打印路径
path.logs: /data/es_logs
 
# Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 0.0.0.0
 
# Set a custom port for HTTP:
http.port: 9200
 
# 节点内存配置
# Lock the memory on startup:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
discovery.zen.minimum_master_nodes: 1
 
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
 
# 开启tcp端口
transport.tcp.compress: true
transport.tcp.port: 9300
 
# 跨域配置
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type
http.cors.allow-credentials: true
 
######## Start Search Guard Demo Configuration ########
# WARNING: revise all the lines below before you go into production
 
# 关闭xpack安全认证
xpack.security.enabled: false
#xpack.monitoring.enabled: false
 
# TLS设置
searchguard.ssl.transport.pemcert_filepath: node1.pem
searchguard.ssl.transport.pemkey_filepath: node1.key
searchguard.ssl.transport.pemkey_password: 7EJJ2hYcJFJQ
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: node1_http.pem
searchguard.ssl.http.pemkey_filepath: node1_http.key
searchguard.ssl.http.pemkey_password: 6eIWAxh4kgHs
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=node1.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
- CN=node2.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
- CN=node3.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
searchguard.authcz.admin_dn:
 - CN=spock.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
# - CN=kirk.yibai.com,OU=Ops,O=yibai Com\, Inc.,DC=yibai,DC=com
 
# 允许演示证书和自动初始化 生产环境上改为false
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
 
# 客户端认证
searchguard.ssl.http.clientauth_mode: OPTIONAL
 
# 启用审计日志记录并将生成的审计跟踪直接存储在Elasticsearch中
searchguard.audit.type: internal_elasticsearch
 
# 允许快照/恢复
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
 
# 可以访问REST的角色
searchguard.restapi.roles_enabled: ["sg_all_access"]
 
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
 
######## End Search Guard Demo Configuration ########

• 在创建es容器的时候，就有部分配置文件，进入容器内可以在config/elasticsearch.yml中看到，然后生成tls证书的时候也会生成一个指定的yml文件，可以在复制好pme和key文件后，把指定节点yml文件的内容复制进去。

至此，凭证修改好了，剩下就是把信息写入到es中了

./sgadmin.sh -cd ../sgconfig/ -icl -nhnv \
  -cacert ../../../config/root-ca.pem \
  -cert ../../../config/spock.pem \
  -key ../../../config/spock.key \
  -keypass 3QgfFoYd8Ken

• 在/tools/out/client-certificates.readme里面取到对应客户端帐号密码

• 注意：所有节点都配置好后，重启哈es节点，然后再执行。如果容器内提示没java环境，可以直接用es自带的jdk，配置下环境变量就可以了。