ES集群搭建
一、安装软件准备
系统：Centos7-2003
软件版本：JDK15.x、ES7.11.X。
官方下载地址：
https://www.elastic.co/cn/downloads/
社区镜像下载
http://dl.elasticsearch.cn/
华为镜像下载
https://mirrors.huaweicloud.com/
Docker容器下载
可选择ES官方镜像或者国内阿里云/网易等镜像
二、操作系统设置
生产环境必须部署在Linux，并需要一些系统配置才可生效。个人环境可以选择Windows，几乎无需配置。无论是单实例还是集群，所有的节点都要以下配置。注意以下配置时，请用root权限账号操作。
2.1 关闭防火墙状态
#关闭命令
service firewalld stop
chkconfig firewalld off
2.2 关闭SELINUX
配置文件：/etc/selinux/config
#SELINUX=enforcing
SELINUX=disabled
2.3 设置hostname
- 若是虚拟机，建议配置独立的主机名，便于区分
- 配置文件：/etc/sysconfig/network
方式一：直接hostnamectl命令
hostnamectl set-hostname elkserver
方式二：配置文件修改
NETWORKING=yes
HOSTNAME=elkserver
2.4 关闭swap
- 配置文件 /etc/sysctl.conf
[root@elkserver /]# vim /etc/sysctl.conf
# 禁用内存与硬盘交换
vm.swappiness=1
# 设置虚拟内存大小
vm.max_map_count=262144
- 执行/sbin/sysctl -p 立即生效
[root@elkserver /]# /sbin/sysctl -p
2.5 文件句柄设置
配置文件： /etc/security/limits.conf
[root@elkserver /]# vim /etc/security/limits.conf
# 进程线程数
* soft nproc 131072
* hard nproc 131072
# 文件句柄数
* soft nofile 131072
* hard nofile 131072
# jvm内存锁定交换
* soft memlock unlimited
* hard memlock unlimited
2.6 ES专用账号并授权
- 创建ES专用账号
useradd es
#授权ES程序目录es账号权限
#假设 ES程序目录、数据目录、日志目录都在/es目录下
chown -R es:es /es/*
2.7 JDK配置
将jdk-15.0.2_linux-x64_bin.tar.gz放到路径/usr/local/src/下并解压
[root@elkserver src]# tar -zxvf jdk-15.0.2_linux-x64_bin.tar.gz
- JDK配置（可选）：/etc/profile
ES最新版本自带jdk版本，默认可以不需要配置，建议配置，便于安装其它java程序辅助
本内容中要求安装JDK15.x
[root@elkserver src]# vim /etc/profile
export JAVA_HOME=/usr/local/src/jdk-15.0.2
export JRE_HOME=$JAVA_HOME/jre
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
- 保存退出后，使配置生效：
[root@elkserver src]# source /etc/profile
[root@elkserver src]# java -version
java version "15.0.2" 2021-01-19
Java(TM) SE Runtime Environment (build 15.0.2+7-27)
Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing)
2.8 JVM临时目录
如在es目录下创建/es/jvm-tmpdir
[root@elkserver es]# mkdir jvm_tmpdir
[root@elkserver elk]# chown -R es:es /es/*
- 方式一：修改环境变量
[root@elkserver src]# vim /etc/profile
export ES_TMPDIR=/es/jvm_tmpdir
[root@elkserver src]# source /etc/profile
- 方式二：在jvm.options中设置（原来就有）
-Djava.io.tmpdir=${ES_TMPDIR}
2.9 JNA临时目录
创建ES临时文件目录：jna_tmpdir，仅限制Linux操作系统
如在es目录下创建/es/jna-tmpdir
在启动JVM中增加此参数
在jvm.options中修改
-Djna.tmpdir=/es/jna_tmpdir
[root@elkserver es]# mkdir jna_tmpdir
[root@elkserver es]# ls
elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 jna_tmpdir jvm_tmpdir kibana-7.11.1-linux-x86_64.tar.gz
三、脚本命令
启动ES脚本
Windows系统：elasticsearch.bat
Linux系统：
当前窗口启动： ./elasticsearch
后台进程启动： ./elasticsearch -d
辅助工具
安全、插件、环境、迁移。。。
四、ES配置文件
elasticsearch.yml 核心配置文件、节点实例属性参数
jvm.options 配置堆栈参数等
log4j2.properties 日志常规配置，默认就好
其他
账户安全/动态生成/keystore安全文件
配置elasticsearch.yml
-
集群名称
默认无需设置，建议设置
如：cluster.name: elk01
-
节点名称
单实例不用设置，集群下必须区分设置，建议设置
如：${HOSTNAME}-9200
-
IP地址
必须设置，默认仅限本地访问
如：network.host: 192.168.237.127
-
访问端口
http端口，默认无需设置
tcp端口，默认需求设置
如：http.port: 9200
transport.port: 9300 -
集群发现
discovery.xxx 单节点无需配置
cluster.xx.xx 单节点默认可无需配置
单节点如：discovery.seed_hosts: [“192.168.237.127:9300”]
cluster.initial_master_nodes: [“192.168.237.127:9300”]
多节点集群如：
discovery.seed_hosts: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]
cluster.initial_master_nodes: [“192.168.237.127:9301”,“192.168.237.127:9302”,“192.168.237.127:9303”]
-
数据目录与日志目录
path.data: /es/es-9200/elk/data
path.logs: /es/es-9200/elk/logs
-
内存交换锁定，此处需要操作系统设置才生效
bootstrap.memory_lock: true
-
防止批量删除索引
action.destructive_requires_name: true -
设置处理器数量4，默认无需设置，单机器多实例需要设置
node.processors: 4 -
设置跨域操作
http.cors.enabled 是否支持跨域，默认为false http.cors.allow-origin 当设置允许跨域，默认为*,表示支持所有域名，如果我们只是允许某些网站能访问，那么可以使用正则表达式。比如只允许本地地址。
如：http.cors.enabled: true
http.cors.allow-origin: “*”
jvm.options
-
JDK 选择
自主配置或者自带
-
GC选择
7.8.1已经采用G1、原则JDK14以上采用G1，以下采用CMS、7.11.x
-
堆栈大小
默认4GB
不超过1/2系统内存
空余1/2闲置内存
内存上限不超过32G，且不能等于32G
-
GC日志
-
gc.log路径设置
五、Kibana安装配置
Kibana作为ES的开发者工具、管理运维工具（不存储数据）。
-
bin
启动脚本、常用工具脚本
-
config
文件配置
-
其它
默认，无需关系
kibana.yml
-
ES配置指向 必须配置，默认仅本地
如：elasticsearch.hosts: [“http://192.168.237.127:9201”]
-
IP+Port设置 IP地址必须配置，默认仅限本机访问
如：server.port: 6601
server.host: “192.168.237.127”
-
元数据索引名称
默认无需配置，建议配置修改
如：kibana.index: “.kibana-elk03”
-
其它设置
安全、语言、其它
六、单节点实战部署
虚拟机：内存8G及以上，处理器4核，硬盘30G
命令 free -m查看系统资源
之前准备工作创建es文件资源目录 mkdir es，进入目录后pwd命令如下
[root@elkserver /]# cd es
[root@elkserver es]# pwd
/es
[root@elkserver es]# hostname
elkserver
在es目录下放入es安装包并解压，
[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz
解压后将解压出的文件夹改名为es-9200
[root@elkserver es]# mv elasticsearch-7.11.1 es-9200
[root@elkserver es-9200]# cd elk
[root@elkserver elk]# mkdir data
[root@elkserver elk]# mkdir logs
[root@elkserver elk]# ls
data logs
同样解压Kibana，并重命名文件夹
[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz
[root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-5601
改变目录权限（上面已提到过）
[root@elkserver es]# chown -R es:es /es/*
进入es-9200文件夹，查看es自带jdk版本
[root@elkserver es-9200]# cd jdk/bin
[root@elkserver bin]# ./java -version
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
进入到es-9200/config/文件夹下进行配置文件的配置
[root@elkserver config]# vim jvm.options
增加如下：
-Xms1g
-Xmx1g
其他不建议修改，如修改比如log目录等。保存后退出 :wq
[root@elkserver config]# vim elasticsearch.yml
详细配置见上面 第四章的内容，附录单节点9200的elasticsearch.yml文件内容如下：（注意将临时文件及日志文件路径事先创建好并授权给es用户）
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk01
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}-9200
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /es/es-9200/elk/data
#
# Path to log files:
#
path.logs: /es/es-9200/elk/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.237.127
#
# Set a custom port for HTTP:
#
http.port: 9200
transport.port: 9300
# Lock the memory on startup:
# #内存交换锁定，此处需要操作系统设置才生效
bootstrap.memory_lock: true
# #
# #防止批量删除索引
action.destructive_requires_name: true
# discovery.type : single-node
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.237.127:9300"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.237.127:9300"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
进入kibana配置路径并修改配置
[root@elkserver es-9200]# cd ..
[root@elkserver es]# ls
elasticsearch-7.11.1-linux-x86_64.tar.gz es-9200 es-9201 es-9202 es-9203 jna-tmpdir kibana-5601 kibana-6601 kibana-7.11.1-linux-x86_64.tar.gz
[root@elkserver es]# cd kibana-5601
[root@elkserver kibana-5601]# ls
bin config data LICENSE.txt node node_modules NOTICE.txt package.json plugins README.txt src x-pack
[root@elkserver kibana-5601]# cd config
[root@elkserver config]
[root@elkserver config]# vim kibana.yml
详细配置见上面 第四章的内容，附录单节点5601的kibana.yml文件内容如下：
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.237.127"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.237.127:9200"]
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana-elk01"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
启动ES
切换到es用户，并回到es-9200目录下进行命令启动
[root@elkserver es-9200]# su es
[es@elkserver es-9200]# ./bin/elasticsearch
-
启动成功后，windows主机浏览器输入es的ip及端口如：http://192.168.237.127:9200/
显示如下，即ES启动成功
{
"name" : "elkserver-9200",
"cluster_name" : "elk01",
"cluster_uuid" : "_o7bbB61RHSVd9kLbTKCiQ",
"version" : {
"number" : "7.11.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a",
"build_date" : "2021-02-15T13:44:09.394032Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
-
检查节点的健康状态
http://192.168.237.127:9200/_cat/health
1621091301 15:08:21 elk01 green 1 1 6 6 0 0 0 0 - 100.0% -
查看节点状态
http://192.168.237.127:9200/_cat/nodes
192.168.237.127 28 88 4 0.04 0.30 0.27 cdhilmrstw * elkserver-9200 -
xshell中关闭 ctrl +c
启动Kibana
进入到es/kibana-5601文件夹下，执行如下命令启动kibana
[es@elkserver kibana-5601]$ ./bin/kibana
-
启动成功后，windows主机浏览器访问kibana的ip及端口如： http://192.168.237.127:5601
-
点击ADD data
-
点击Sample data 依次添加样例数据（样例数据很重要，后续会使用）
-
进入左上角菜单 Management——>Stack Monitoring
-
点击 Or, set up with self monitoring 使用内部手机设置
-
打开 Turn on monitoring
-
-
如看到es节点变黄，则在kibana中的Dev tools中执行如下即可
PUT /_settings { "number_of_replicas": 0 }xshell中关闭 ctrl +c
七、集群环境搭建
ES集群配置
Elastic集群模式必须至少2个实例以上，一般建议3个节点以上，可以保障其中一个节点失效，集群仍然可以服务。集群模式与单实例模式大部分配置上是一样的，仅需修改集群通信差异部分。
集群环境说明
在这里在一台机器但起3个节点服务的集群。原则上与三台机器各起一个节点服务是一样的。
服务器IP：192.168.237.127
服务器hostname：elkserver
ES文件夹目录：es/es-9201、es/es-9202、es/es-9203
三个服务的es端口号：9201、9202、9203
Kibana启动一个节点即可：es\ kibana-6601 端口号：6601
安装说明
注意需重新解压一个es并命名为es-9201
[root@elkserver es]# tar -zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz
[root@elkserver es]# mv elasticsearch-7.11.1 es-9201
注意需重新解压一个kibana并命名为 kibana-6601
[root@elkserver es]# tar -zxvf kibana-7.11.1-linux-x86_64.tar.gz
[root@elkserver es]# mv kibana-7.11.1-linux-x86_64 kibana-6601
ES的9201配置文件内容如下：
jvm.options
################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html
## for more information.
##
################################################################
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
-Xms1g
-Xmx1g
################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################
## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
附es-9201节点的elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk03
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}-9201
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /es/es-9201/elk/data
#
# Path to log files:
#
path.logs: /es/es-9201/elk/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
# Lock the memory on startup:
#内存交换锁定，此处需要操作系统设置才生效
bootstrap.memory_lock: true
#
#防止批量删除索引
action.destructive_requires_name: true
#
#设置处理器数量4，默认无需设置，单机器多实例需要设置
node.processors: 4
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.237.127
#
# Set a custom port for HTTP:
#
http.port: 9201
transport.port: 9301
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
- 创建日志路径及数据路径
[root@elkserver es-9201]# mkdir elk
[root@elkserver es-9201]# ls
bin config elk jdk lib LICENSE.txt logs modules NOTICE.txt plugins README.asciidoc
[root@elkserver es-9201]# cd elk
[root@elkserver elk]# mkdir data
[root@elkserver elk]# mkdir logs
[root@elkserver elk]# ls
data logs
[root@elkserver elk]# chown -R es:es /es/*
es-9202、es-9203文件夹的生成方法
- 其余两个节点，可用第一个节点文件夹复制的形式，再修改配置。
[root@elkserver es]# cp -r es-9201/ es-9202
[root@elkserver es]# cp -r es-9201/ es-9203
- 只需修改两个节点的elasticsearch.yml 以及临时文件和日志文件的名称即可。
附es-9202节点的elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk03
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}-9202
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /es/es-9202/elk/data
#
# Path to log files:
#
path.logs: /es/es-9202/elk/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
# Lock the memory on startup:
#内存交换锁定，此处需要操作系统设置才生效
bootstrap.memory_lock: true
#
#防止批量删除索引
action.destructive_requires_name: true
#
#设置处理器数量4，默认无需设置，单机器多实例需要设置
node.processors: 4
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.237.127
#
# Set a custom port for HTTP:
#
http.port: 9202
transport.port: 9302
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
附es-9203节点的elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk03
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}-9203
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /es/es-9203/elk/data
#
# Path to log files:
#
path.logs: /es/es-9203/elk/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
# Lock the memory on startup:
#内存交换锁定，此处需要操作系统设置才生效
bootstrap.memory_lock: true
#
#防止批量删除索引
action.destructive_requires_name: true
#
#设置处理器数量4，默认无需设置，单机器多实例需要设置
node.processors: 4
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.237.127
#
# Set a custom port for HTTP:
#
http.port: 9203
transport.port: 9303
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.237.127:9301","192.168.237.127:9302","192.168.237.127:9303"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
- 注意：新生成文件夹后需要进行重新授权。chown -R es:es /es/*
启动ES集群环境
- 切换到es账户下 su - es，分别依次启动三个节点的es服务
[es@elkserver es-9201]$ ./bin/elasticsearch
[es@elkserver es-9202]$ ./bin/elasticsearch
[es@elkserver es-9203]$ ./bin/elasticsearch
检查三个节点启动是否成功
- 依次在windows主机的浏览器中输入三个ES节点的地址信息进行查看
http://192.168.237.127:9201/
http://192.168.237.127:9202/
http://192.168.237.127:9203/
检查三个节点的健康状态
http://192.168.237.127:9201/_cat/health
结果：1621074718 10:31:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
http://192.168.237.127:9202/_cat/health
结果：1621074689 10:31:29 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
http://192.168.237.127:9203/_cat/health
结果：1621074658 10:30:58 elk03 green 3 3 53 25 0 0 0 0 - 100.0%
检查集群是否成功启动
-在任意节点看即可
http://192.168.237.127:9202/_cat/nodes
结果：
192.168.237.127 39 97 7 0.28 1.19 0.88 cdhilmrstw * elkserver-9201
192.168.237.127 26 97 7 0.28 1.19 0.88 cdhilmrstw - elkserver-9203
192.168.237.127 43 97 6 0.28 1.19 0.88 cdhilmrstw - elkserver-9202
配置Kibana
附6601节点Kibana的kibana.yml配置
- 注elasticsearch.hosts只需配置集群中一个节点即可
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 6601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.237.127"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.237.127:9201"]
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana-elk03"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
启动Kibana环境
[es@elkserver kibana-6601]$ ./bin/kibana
检查Kibana环境启动是否成功
- windows主机浏览器访问：
http://192.168.237.127:6601/
-
启动成功后，windows主机浏览器访问kibana的ip及端口如： http://192.168.237.127:6601
-
点击ADD data
-
点击Sample data 依次添加样例数据（样例数据很重要，后续会使用）
-
进入左上角菜单 Management——>Stack Monitoring
-
点击 Or, set up with self monitoring 使用内部手机设置
-
打开 Turn on monitoring
-
-
如看到es节点变黄，则在kibana中的Dev tools中执行如下即可
PUT /_settings { "number_of_replicas": 0 }xshell中关闭 ctrl +c
- 安装部署成功！
258
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
