ELK系列（一） ElasticSearch 8.9.2集群搭建

已于 2024-01-18 18:03:29 修改
阅读量819 收藏 11
点赞数 10
分类专栏： ELK 文章标签： elk elasticsearch
于 2023-12-29 11:36:35 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/BourneSu/article/details/135285691
版权

ELK系列传送门

ELK系列（一） ElasticSearch 8.9.2集群搭建

ELK系列（二）Kafka集群3.4.0搭建

基础环境配置

主机规划

主机名     内网地址      搭建程序

ELK8-1   192.168.0.1  elasticsearch+kafka+zookeeper
ELK8-2   192.168.0.2  elasticsearch+kafka+zookeeper
ELK8-3   192.168.0.3  elasticsearch+kafka+zookeeper
               192.168.0.4  kibana+logstash

环境配置

ELK8-1 ELK8-2 ELK8-3 kibana+logstash 统一配置

Base源及epel源配置 关闭防火墙
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum -y install vim wget lvm2 tcping
wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo
iptables -F
iptables -X
iptables -Z
systemctl stop firewalld

java环境配置
tar xf jdk-20_linux-x64_bin.tar.gz -C /usr/local/
vim /etc/profile.d/java.sh
JAVA_HOME=/usr/local/jdk-20.0.1
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH

source /etc/profile.d/java.sh
[root@ELK8-1 jdk-20.0.1]# java -version
java version "20.0.1" 2023-04-18
Java(TM) SE Runtime Environment (build 20.0.1+9-29)
Java HotSpot(TM) 64-Bit Server VM (build 20.0.1+9-29, mixed mode, sharing)

lvm搭建--数据目录/data

所需操作主机：ELK8-1 ELK8-2 ELK8-3

LVM挂载数据盘到/data目录
fdisk /dev/vdb
pvcreate /dev/vdb1
vgcreate vg1 /dev/vdb1
lvcreate -l 25599 -n lv1 vg1
mkfs.xfs /dev/vg1/lv1
mkdir /data
mount /dev/vg1/lv1 /data
mkdir /data/{elasticsearch,kafka,zookeeper}

ES集群搭建

安装yum源

所需操作主机：ELK8-1 ELK8-2 ELK8-3 logstash1

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

开启elastic-8.x版本下载
vi /etc/yum.repos.d/elastic.repo
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装ES
yum install --enablerepo=elasticsearch elasticsearch

ELK8-1 ELK8-2 ELK8-3 安装后会得到超管用户elastic和密码请保存

ELK8-1  192.168.0.1  tIxjmD8nWabee6Rs0QRm
ELK8-2  192.168.0.2  1FL54xSW8N8sqP5o2RRH
ELK8-3  192.168.0.3  enxjhT8UFylO6eVqGEbp

系统优化设置

所需操作主机：ELK8-1 ELK8-2 ELK8-3

守护进程配置
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

jvm设置
sed -i 's/-Xms4g/-Xms4g/' /etc/elasticsearch/jvm.options
sed -i 's/-Xmx4g/-Xmx4g/' /etc/elasticsearch/jvm.options

data目录设置
mkdir -p /data/elasticsearch/data 
mkdir -p /data/elasticsearch/logs 
chown -R elasticsearch:elasticsearch /data/elasticsearch

所需操作主机：ELK8-1 ELK8-2 ELK8-3 logstash1

系统参数设置
vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p

ELK8-1启动ElasticSearch

ELK8-1 作为集群节点 ELK8-2及ELK8-3 加入ELK8-1所在集群

[root@ELK8-1 ~]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK8  #集群名
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ELK8-1  #节点名
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch/data  #数据目录
#
# Path to log files:
#
path.logs: /data/elasticsearch/logs  #日志目录
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.0.1  #内网地址
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200  #默认端口
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.0.1"]  #可被发现节点
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.0.1"]  #集群可成为master的节点
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 11-09-2023 06:20:32
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true  #开启xpack

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:  #ssl证书配置 http.p12yum安装默认生成
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl: #内网节点ssl配置 transport.p12安装默认生成
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["ELK8-1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0  内网建议注释

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------


启动ELK8-1
systemctl start elasticsearch
查看EL8-1状态及节点状态
curl -k -u elastic:tIxjmD8nWabee6Rs0QRm https://192.168.0.1:9200
curl -k -u elastic:tIxjmD8nWabee6Rs0QRm https://192.168.0.1:9200/_cat/nodes
192.168.0.1 84 98 3 0.09 0.07 0.09 cdfhilmrstw * ELK8-1 *号代表主节点

ELK8-2 ELK8-3加入集群

ELK8-1生成token

[root@ELK8-1 data]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node  
eyJ2ZXIiOiI4LjkuM7RuSa27ilvkGSBwiL4NB1vRxnGNqGhjKKEeJ2DinW8diovGp0d2YYTqCIqS8iUTWozwo1iSZTbGDN0e9Uido7lhUhJN1edsc7ZcHI5THZ3In0=

ELK8-2 ELK8-3 通过token加入集群
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjkuMiIsImFkciI6WyIxODMuMjUwLjE3OS43Mzo5MjAwIl0sImZnciI6ImM2OGM0YjM0OWUyYWFjMmViODhjY2NiMDhmMjRhMmFiNmY1MjMxYTllMDg4ODAzYTgwZWIwY2I1NzAxZDgxZGIiLCJrZXkiOiIwSDRLZzRvQmRCTXJvYkUwTEV4UzpqYmFCUFZVWlRBcWNiZjVCNHI5THZ3In0=

ELK8-2 ELK8-3 配置文件修改  ELK8-3配置参考ELK8-2
[root@elk8-2 ~]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK8
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ELK8-2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch/data
#
# Path to log files:
#
path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.0.2
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 11-09-2023 07:04:53
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Discover existing nodes in the cluster
discovery.seed_hosts: ["192.168.0.1:9300"] 

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

ELK8-2 ELK8-3启动elasticsearch
systemctl start elasticsearch

ELK8-1观察节点状态 ELK8-2 ELK8-3已加入集群
[root@ELK8-1 logs]# curl -k -u elastic:tIxjmD8nWabee6Rs0QRm https://192.168.0.1:9200/_cat/nodes  
192.168.0.2  9 82 0 0.03 0.13 0.12 cdfhilmrstw - ELK8-2
192.168.0.1 19 81 0 0.03 0.02 0.05 cdfhilmrstw * ELK8-1
192.168.0.3  6 82 9 0.34 0.24 0.13 cdfhilmrstw - ELK8-3

ElasticSearch集群搭建完成

诉光海
关注
  • 10
    点赞
  • 踩
  • 11
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
ELK(八)ElasticSearch集群
weixin_43744059的博客
09-23 216
集群节点 ElasticSearch集群是由多个节点组成的,通过cluster.name设置集群名称,并且用于区分其他的集群,每个节点通过node.name指定节点的名称 在ElasticSearch中,节点的类型主要有4种 master节点 配置文件中node,master属性为true,就有资格被选为master节点 master节点用于控制整个集群的操作,比如创建或删除索引,管理其他非m...
ELK日志中心搭建（一）-elasticsearch8.5.3
03-04
elasticsearch-8.5.3-linux-x86_64.tar.gz filebeat-8.5.3-linux-x86_64.tar.gz kafka_2.12-3.4.0.tgz logstash-8.5.3-linux-x86_64.tar.gz
elk集群部署手册
08-22
elasticsearch集群，ELK(日志分析系统)： Elasticsearch + Logstash + Kibana集群环境部署及应用
ELK8.4+kafka日志收集系统离线部署教程，巨详细，适合新手
Vic的博客
01-16 347
网上的教程很少提到安全相关的配置，在网络安全越注重的今天，安全配置也不容忽视，不要为了方便而把安全有关的东西都禁掉。好了，Filebeat日志收集就介绍这么多，在实际使用的时候可以根据自己的业务需求来，在相应的服务器上安装filebeat，然后配置好相应的filebeat.yml文件，不同的应用，不同的日志类型，所编写的yml文件不同得根据实际灵活变通，logstash的日志收集配置文件也是，针对具体应用的日志收集写法，因篇幅有限，这里不一一列举，可以百度找一些别人收集的日志的模板改成自己的。
ELK v7.8集群部署
weixin_45203131的博客
09-07 510
ElasticSearch基本概念 Node：运行单个ES实例的服务器. Cluster：一个或多个节点构成集群 Index：索引是多个文档的集合 Document：Index里每条记录称为Document，若干文档构建一个Index Type：一个Index可以定义一种或多种类型，将Document逻辑分组 Field：ES存储的最小单元 Shards：ES将Index分为若干份，每一份就是一个分片 Replicas：Index的一份或多份副本。 环境配置 组件名称 ip 内存 c
运维入坑实践之ELK集群部署
ding_zk的博客
05-10 482
版本说明 系统版本： CentOS Linux release 7.4.1708 (Core) 软件版本： jdk1.8 Elasticsearch：7.12.1 Kibana：7.12.1 Logstash：7.12.1 角色分配 ip规划 角色分配 192.168.25.101 ES:master、datanode Kibana 192.168.25.102 ES:master、datanode Logstash .
@ELK集群环境部署搭建
ଲ一笑奈@何
07-01 777
elk部署
ELK集群部署---Kibana的部署
NancyLCL的博客
12-23 1235
ELK集群部署---Kibana的部署 最后选择一个默认展示的索引日志即可。将ES上的三个索引全部创建。
ELK系列（三） Kibana8.9.2搭建
BourneSu的博客
01-04 429
ELK系列（三） Kibana8.9.2搭建。
ELK系列（四） Logstash8.9.2搭建
BourneSu的博客
01-16 470
ELK系列（四） Logstash8.9.2搭建
ELK集群部署（elasticsearch\logstash\kibana）
01-18
ELK日志收集集群部署软件包，全rpm包，共四个。jdk1.8\es\logstash\kibana
ELK集群搭建手册1
08-08
# cd /etc/elasticsearch/# lltotal 28-rw-rw---- 1 root elasticsearch 207 Nov 5 11
ElasticSearch7.x集群搭建(es7)主从读写分离搭建教程
06-16
适用人群 开发从业人员，大数据从业人员...课程概述 ElasticSearch7.x集群？Elasticsearch-head7.x官网都没有的插件整合？cerebro插件酷炫暗黑插件神器？一次搞定 官网版本 主从读写 中文分词 Head插件 cerebro插件
docker、rpm方式 安装和部署 elk 8.1.0的过程(elasticsearch + kibana + logstash)
festone000的专栏
03-12 7852
docker、rpm 方式部署elk(elasticsearch、kibana等) 8.1.0 ，以及logstash等的使用。
Linux 使用 yum安装 ELK服务，yum 安装elasticsearch和Kibana(未写完)
雪花凌落的盛夏
05-27 661
是一个近实时（NRT）的分布式搜索和分析引擎，它可以用于全文搜索，结构化搜索以及分析。它是一个建立在全文搜索引擎 Apache Lucene 基础上的搜索引擎，使用 Java 语言编写。它是一个具有近实时（NRT）传输能力的数据收集、过滤、分析引擎，用来进行数据收集、解析、过滤，并最终将数据发送给ES。它是一个为 ElasticSearch 提供分析和展示的可视化 Web 平台。它可以在 ElasticSearch 的索引中查找，交互数据，并生成各种维度表格、图形以及仪表盘。
ELK 使用 metricbeat监控数据
最新发布
weixin_44295677的博客
05-30 584
使用metricbeat监控elastic
【PB案例学习笔记】-09滚动条使用
再小的帆也能远航，把分享变成一种习惯
05-27 456
这是PB案例学习笔记系列文章的第8篇，该系列文章适合具有一定PB基础的读者。通过一个个由浅入深的编程实战案例学习，提高编程技巧，以保证小伙伴们能应付公司的各种开发需求。文章中设计到的源码，小凡都上传到了gitee代码仓库需要源代码的小伙伴们可以自行下载查看，后续文章涉及到的案例代码也都会提交到这个仓库【如果对小伙伴有所帮助，希望能给一个小星星⭐支持一下小凡。
elk kafka 集群搭建
07-16
要搭建ELK和Kafka集群，可以按照以下步骤操作： 1.安装Java 首先要安装Java环境，因为ELK和Kafka都是基于Java开发的，所以需要先安装Java环境。 2.安装Elasticsearch Elasticsearch是ELK中的组件之一，它是一个分布式的开源搜索和分析引擎，用于存储和处理日志数据。安装Elasticsearch的过程可以参考官方文档。 3.安装Logstash Logstash是ELK中的另一个组件，它用于收集、处理和转发日志数据。安装Logstash的过程可以参考官方文档。 4.安装Kibana Kibana是ELK中的第三个组件，它是一个基于Web的界面，用于可视化和分析日志数据。安装Kibana的过程可以参考官方文档。 5.安装Kafka Kafka是一个基于分布式的发布-订阅消息系统，它可以处理大量的实时数据流。安装Kafka的过程可以参考官方文档。 6.配置集群 配置集群需要设置各个组件之间的通信方式和节点信息，具体的配置可以参考各个组件的官方文档。 7.测试集群 完成集群的配置后，可以通过发送一些消息来测试集群是否正常工作。 以上是ELK和Kafka集群搭建的大致步骤，具体的操作细节可以参考各个组件的官方文档。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值