1.ElasticSearch系列之集群部署及权限认证

算法小生Đ

已于 2024-06-20 21:13:58 修改

阅读量583

点赞数

分类专栏： NOSQL 文章标签： elasticsearch 大数据 java

于 2022-10-18 20:19:22 首次发布

本文链接：https://blog.csdn.net/SJshenjian/article/details/127395865

版权

NOSQL 专栏收录该内容

25 篇文章 1 订阅

订阅专栏

一、集群部署

第一步：安装JDK

JDK要求jdk1.8+，不安装也可以，ES自带JDK

第二步：系统配置

2.1 禁用交换区

    sudo swapoff -a

2.2 开最大文件数的限制

    编辑文件 /etc/security/limits.conf把nofile设置为65536
    或者执行
    echo  "* soft nofile 65536" >> /etc/security/limits.conf
    echo  "* hard nofile 65536" >> /etc/security/limits.conf
    echo  "elasticsearch soft memlock unlimited" >> /etc/security/limits.conf
    echo  "elasticsearch hard memlock unlimited" >> /etc/security/limits.conf

2.3 设置虚拟内存

    临时生效 sysctl -w vm.max_map_count=262144
    永久生效 编辑/etc/sysctl.conf vm.max_map_count
    验证命令 sysctl vm.max_map_count

2.4 线程数量

    root用户执行 ulimit -u 4096
    或者
    编辑/etc/security/limits.conf把nproc设置为4096
    echo  "* soft nproc 4096" >> /etc/security/limits.conf
    echo  "* hard nproc 4096" >> /etc/security/limits.conf

第三步：创建用户并设置密码

    useradd elasticsearch
    passwd elasticsearch

第四步：安装目录介绍

4.1 创建安装包目录

mkdir -p home/software

4.2 上传压缩包并解压

tar zxvf ES安装包.tar.gz
tar xf elasticsearch-7.6.2-linux-x86_64.tar.gz -C /usr/local
tar xf kibana-7.6.2-linux-x86_64.tar.gz -C /usr/local

4.3 数据日志位置介绍及创建

ES安装目录 /usr/local
ES_HOME /usr/local/elasticsearch-7.6.2
数据目录 /home/elasticsearch/data
日志目录 /home/elasticsearch/logs
配置目录 /home/elasticsearch/config

mkdir /home/elasticsearch/data /home/elasticsearch/logs /home/elasticsearch/config

4.4 拷贝config目录

cp /usr/local/elasticsearch-7.6.2/config/* /home/elasticsearch/config

第五步：修改ES基本配置

5.1 进入elasticsearch.yml

vi /usr/local/elasticsearch-7.6.2/config/elasticsearch.yml

5.2 修改elasticsearch.yml配置

# 集群名称
cluster.name: shenjian
# 配置数据目录
path.data: /home/elasticsearch/data
# 配置日志目录
path.logs: /home/elasticsearch/logs
# 启动时是否锁定内存
bootstrap.memory_lock: true
# 节点启动时要加入的集群机器列表，可配置所有节点
discovery.seed_hosts: ["192.168.0.6","192.168.0.7", "192.168.0.8"]
# 初始化可成为主节点的列表
cluster.initial_master_nodes: ["192.168.0.6","192.168.0.7", "192.168.0.8"]

5.3 修改jvm.options配置

-Xms20g
-Xmx20g

# JAVA8 9 GC LOG路径
8:-Xloggc:/home/elasticsearch/logs/gc.log
file=/home/elasticsearch/logs/gc.log

第六步：创建启动脚本

start_elasticsearch.sh

###指定ES的安装目录
ES_HOME="/usr/local/elasticsearch-7.6.2"
###指定ES配置文件的目录
export ES_PATH_CONF="/home/elasticsearch/config"
pid=`ps ax | grep elasticsearch | grep java | head -1 | awk '{print $1}'`
kill -9 $pid
echo "kill -9 $pid success"
ulimit -u 4096
#IP=`ifconfig | grep "inet addr:" |grep 192.168.0| sed '/127/d' | awk '{print $2}' | awk -F : '{print $2}'`
#echo $IP
IP=192.168.0.6
$ES_HOME/bin/elasticsearch -d -Enode.name=HOST${IP} -Enetwork.host=${IP}

第七步：授权并启动

chown -R elasticsearch:ealsticsearch /home/elasticsearch
chmod -x /home/elasticsearch/start_elasticsearch.sh
./start_elasticsearch.sh

第八步：验证集群启动情况

curl http://192.168.0.8:9200/_cat/nodes?pretty

可以看到打印信息
192.168.0.7 10 78  0 0.11 0.05 0.06 dilm * HOST192.168.0.7
192.168.0.8 17 79  1 0.02 0.02 0.05 dilm - HOST192.168.0.8
192.168.0.6 16 87 29 0.77 0.38 0.18 dilm - HOST192.168.0.6

遇到的问题

低版本的Centos抛出的警告

java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
	at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:341) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:616) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:258) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
	
	解决方案：
	
	修改elasticsearch.yml 添加一下内容
    bootstrap.memory_lock: false
    bootstrap.system_call_filter: false

java.io.IOException: No route to host

Caused by: java.io.IOException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:688) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
	
	192.168.0.6机器出现该问题，查看防火墙，发现与其他两台不一致
	firewall-cmd --state # 查看防火墙状态
    systemctl stop firewalld.service # 关闭防火墙
    systemctl disable firewalld.service # 禁止开机自启动
    ok,解决

二、权限认证

1. 在master节点上创建秘钥库

export ES_PATH_CONF="/home/elasticsearch/config" &&   /usr/local/elasticsearch-7.6.2/bin/elasticsearch-keystore create

2. 在所有节点中开启ssl认证

2.1 生成elastic-stack-ca.p12

/usr/local/elasticsearch-7.6.2/bin/elasticsearch-certutil ca --days 36500

## 出现Please enter the desired output file [elastic-stack-ca.p12]:输入全路径  
/home/elasticsearch/config/certs/elastic-stack-ca.p12

2.2 生成证书：elastic-certificates.p12

/usr/local/elasticsearch-7.6.2/bin/elasticsearch-certutil cert --days 36500 --ca /home/elasticsearch/config/certs/elastic-stack-ca.p12

## 出现 Enter password for CA (/home/elasticsearch/config/certs/elastic-stack-ca.p12) : 数据刚才设置的证书密码
后出现Please enter the desired output file [elastic-certificates.p12]:，输入全路径
 /home/elasticsearch/config/certs/elastic-certificates.p12

2.3 将elasticsearch节点密码添加至elasticsearch-keystore

/usr/local/elasticsearch-7.6.2/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

/usr/local/elasticsearch-7.6.2/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

2.4 将两个证书一个密码文件等拷贝至集群其他节点

scp -r /home/elasticsearch/config/certs elasticsearch@192.168.0.8:/home/elasticsearch/config/
scp /home/elasticsearch/config/elasticsearch.keystore elasticsearch@192.168.0.8:/home/elasticsearch/config/

2.5 在所有节点elasticsearch.yml中新增配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path:  certs/elastic-certificates.p12

3. 在master节点上设置密码

export ES_PATH_CONF="/home/elasticsearch/config"  &&  IP="192.168.0.7" && name="HOST192.168.0.7" && /usr/local/elasticsearch-7.6.2/bin/elasticsearch-setup-passwords interactive  -Enode.name="HOST192.168.0.7" -Enetwork.host="192.168.0.7"

注意：1.此操作会对elasticsearch、logstash、kibana分别设置登录密码（默认es用户名为elastic，
logstash用户名为logstash_system，kibana用户名为kibana）.

2.这些内置用户存储在.security由X-Pack安全性管理的特殊索引中。如果更改了密码或禁用了用户，则该更改将自动反映在群集中的每个节点上。这也意味着如果您的.security索引被删除或从快照恢复，那么您应用的任何更改都将丢失。

4. 验证权限

4.1 未输入用户

请求：curl http://192.168.0.8:9200/_cat/nodes?pretty
响应：{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

4.2 输入用户及密码

curl -u elastic http://192.168.0.8:9200/_cat/nodes?pretty
Enter host password for user 'elastic':
192.168.0.7 10 78 6 0.02 0.13 0.13 dilm * HOST192.168.0.7
192.168.0.8  9 77 4 0.00 0.08 0.10 dilm - HOST192.168.0.8

4.3 证书有效期查看

http://IP:9200/_ssl/certificates?pretty
在这里插入图片描述

OK,到此权限配置成功

欢迎关注公众号算法小生或沈健的技术博客shenjian.online

算法小生Đ

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
1.ElasticSearch系列之集群部署及权限认证

ES安装目录 /usr/local数据目录 /home/elasticsearch/data日志目录 /home/elasticsearch/logs配置目录 /home/elasticsearch/config。
复制链接

扫一扫