ELK搭建（Linux&Windows）
☁️前言
ELK + kafka(zookeeper) + filebeat 的设计模型
ELK搭建 – win10环境
提前准备
官网下载最新的windows版本zip包
ELK+filebeat官网下载地址
网盘链接：https://pan.baidu.com/s/1nK8wVBbvvk9TH7rZBiGzOQ
提取码：YYDS
版本为7.5.2 系统默认使用JDK11.所以需要将环境改为JDK11
JDK 网盘地址
链接：https://pan.baidu.com/s/1fnLxbzRsQ6ziwO9JCBKRqg
提取码：YYDS
安装使用
安装elasticsearch
首先解压elasticsearch-7.5.2-windows-x86_64.zip，命令行执行启动命令
启动之后浏览器访问：http://localhost:9200/
安装kibana
解压kibana-7.5.2-windows-x86_64.zip，修改配置文件
修改…\kibana-7.5.2-windows-x86_64\config下的kibana.yml文件。增加配置
server.port: 5601
server.name: "DESKTOP-IFO48BG"
server.host: "localhost"
elasticsearch.hosts: "http://localhost:9200/"
命令行启动服务
ELK系统添加x-pack安全插件
x-pack是一个Elastic Stack扩展，它提供安全性、警报、监视、报告、机器学习和许多其他功能
生成证书
在…\ELK\elasticsearch-7.5.2\bin> 目录下生成证书
a) 输入命令./elasticsearch-certutil ca生成证书
b) 输入证书保存路径和密码
c) 输入命令./elasticsearch-certutil cert --ca .\config\elastic-stack-ca.p12, 生成第二个证书（注意命令最后参数是第一个证书的相对路径）
d) 输入第一个证书密码进行验证
e) 输入第二个证书保存的路径和密码
将elasticsearch节点密码添加至elasticsearch-keystore
a) 输入命令
./elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
b) 输入第二个证书的密码进行验证
c) 输入命令
./elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
d) 输入第二个证书的密码进行验证
修改elasticsearch各个节点config目录下的elasticsearch.yml文件
添加以下配置属性，注意证书的相对路径
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: .\elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: .\elastic-certificates.p12
重启elasticsearch
待elasticsearch所有节点重启完毕后，设置elasticsearch 节点密码
a)执行设置密码命令
./elasticsearch-setup-passwords interactive
b)按PowerShell提示操作，按y确定继续操作
c)设置各个系统预设users密码（为了方便记忆可以设置成一样的，后面在kibana可以进行修改）
修改kibana配置
a) 停止kibana服务
b) 进入kibana config目录修改kibana.yml档，添加以下属性配置（注意密码是设置节点elastic用户名的密码）
elasticsearch.username: "elastic"
elasticsearch.password: "密码"
c) 重启kibana
20220212更新 安装7.17.0版本，适配IK分词器7.17版本
安装elasticsearch
首先解压elasticsearch-7.17.0-windows-x86_64.zip，命令行执行启动命令
启动之后浏览器访问：http://localhost:9200/
安装kibana
解压kibana-7.17.0-windows-x86_64.zip，修改配置文件
修改…\kibana-7.17.0-windows-x86_64\config下的kibana.yml文件。增加配置
server.port: 5601
server.name: "DESKTOP-IFO48BG"
server.host: "localhost"
elasticsearch.hosts: "http://localhost:9200/"
命令行启动服务
注意
本次安装在虚拟机的windows环境，发现安装之后ELK只能用localhouse访问，不能用IP访问。 解决这个问题，需要做一些配置1.在elasticsearch 中的config下的elasticsearch .yml文件新增配置
http.host: 0.0.0.0
network.host: 0.0.0.0
discovery.seed_hosts: ["0.0.0.0", "[::1]"]
2.在kibana config下的kibana.yml文件
server.host: "localhost"
改为
server.host: "0.0.0.0"
ELK搭建 – Docker中Linux环境
提前准备
首先需要下载ELK3件套+kafka中间件+filebeat的tar包（本文使用7.5.1 版本
ELK+filebeat官网下载地址
Kafka官网下载地址
☀️ 搭建步骤
所有机器都关闭防火墙和selinux
systemctl stop firewalld
systemctl disable firewalld
vim /etc/sysconfig/selinux
SELinux=disabled
setenforce 0
安装elasticsearch
- 将下载好的tar包上传解压安装
因为es7.2及以上版本要求java11及以上的安装环境，从官网查询得知，es自带了java11，所以我们只要把我们本地的java环境变量给注释掉重启即可。如果该机器上确实又有其他项目需要用到java8怎么办，那么就直接修改es配置让它使用自己自带的java11。
vim bin/elasticsearch 需要在前面加这javahome目录这一行。
JAVA_HOME='/usr/local/elasticsearch-7.5.1/jdk'
source "`dirname "$0"`"/elasticsearch-env
- 配置启动
cd /usr/local/elasticsearch-7.5.1/config
vim elasticsearch.yml 需要配置下面这些项
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: IMP
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch/data
#
# Path to log files:
#
path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: false
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["127.0.0.1", "[::1]"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
- 创建日志和数据目录
mkdir -p /data/elasticsearch/{data,log}
- 启动
cd /usr/local/elasticsearch-7.5.1/bin;
./elasticsearch -d
安装kibana
- 上传安装tar包
- 配置启动
cd /usr/local/kibana-7.5.1-linux-x86_64
vi config/kibana.yml
注意： elasticsearch的地址要写上面安装好的elasticsearch的地址
server.port: 5601
server.host: “0.0.0.0”
elasticsearch.hosts©: [“http://localhost:9200”]
配置中文
i18n.locale: “zh-CN”
- 启动
cd /usr/local/kibana-7.5.1-linux-x86_64/bin/
nohup ./kibana &
- 验证
访问192.168.9.168:5601，会进入kibana的系统页面。
安装kafka+zookeeper
kafka的包里面就自带了zookeeper，所以不需要额外的下载
cd /usr/local/src
tar -xzvf kafka_2.12-2.3.1.tgz -C /usr/local
- 安装配置zk
mkdir -p /data/zookeeper/{data,logs}
vi /usr/local/kafka_2.12-2.3.1/config/zookeeper.properties
Zookeeper 配置如下
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10
- 启动zk
Java配置生效
source /etc/profile
nohup /usr/local/kafka_2.12-2.3.1/bin/zookeeper-server-start.sh /usr/local/kafka_2.12-2.3.1/config/zookeeper.properties > /data/zookeeper/logs/zk_output.log 2>&1 &
- 验证zk
- 配置kafka
cd /usr/local/kafka_2.12-2.3.1/config/
vi server.properties
配置如下
broker.id=0
listeners=PLAINTEXT://192.18.2.15:9092
num.network.threads=3
num.io.threads=8
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka-logs-server
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
- 启动kafka
source /etc/profile
nohup /usr/local/kafka_2.12-2.3.1/bin/kafka-server-start.sh /usr/local/kafka_2.12-2.3.1/config/server.properties > /data/kafka/logs/run_output.log 2>&1 &
- 验证Kafka
- 单独验证Kafka是否通行
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
bin/kafka-topics.sh -list -zookeeper localhost:2181
bin/kafka-console-producer.sh --broker-list 192.18.2.15:9092 --topic test
bin/kafka-console-consumer.sh --bootstrap-server 192.18.2.15:9092 --topic test --from-beginning
安装logstash
- 下载logstash并解压
cd /usr/local/src
tar -xzvf logstash/logstash-7.5.1.tar.gz -C /usr/local
- 编辑配置文件
cd /usr/local/logstash-7.5.1/config/
vi cyk_logstash.conf
input {
kafka {
bootstrap_servers => ["192.18.2.15:9092,192.18.2.15:9093"]
topics => "topic_cyk_filebeat_config_output.kafka_cyk215"
codec => "json"
}
}
output {
if "cas" in [tags]{
elasticsearch {
hosts => ["192.168.9.168:9200"]
index => "cas-%{+YYYY.MM.dd}"
}
}
if "uum" in [tags]{
elasticsearch {
hosts => ["192.168.9.168:9200"]
index => "uum-%{+YYYY.MM.dd}"
}
}
}
- 启动logstash
su - elastic
/usr/local/logstash-7.5.1/bin/logstash -f /usr/local/logstash-7.5.1/config/cyk_logstash.conf &
安装Filebeat搜集日志
- 下载filebeat并解压
cd /usr/local/src
tar -xzvf filebeat-7.5.1-linux-x86_64.tar.gz -C /usr/local
cd /usr/local/ && mv filebeat-7.5.1-linux-x86_64/ filebeat-7.5.1
- 配置filebeat
先把filebeat的目录更改一下属主 chown elastic:root -R /usr/local/filebeat-7.5.1
cd /usr/local/filebeat-7.5.1/
vi cyk_filebeat.yml
filebeat.inputs:
- type: log
tail_files: true
paths:
- /opt/apache-tomcat-6.0.39/logs/catalina.out
fields:
appid: appid_cyk_filebeat_config_tag_input_cyk215
tags: ["uum"]
output.kafka:
hosts: ["192.18.2.15:9092","192.18.2.15:9093"]
topic: 'topic_cyk_filebeat_config_output.kafka_cyk215'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
- 启动filebeat
su - elastic
/usr/local/logstash-7.5.1/bin/logstash -f /usr/local/logstash-7.5.1/config/cyk_logstash.conf &
ELK同步Mysql中的表数据
准备链接驱动，并上传
在Logstash中配置JDBC相关配置
vi cyk_logstash.conf
input{
kafka {
bootstrap_servers => ["192.18.2.15:9092,192.18.2.15:9093"]
topics => "topic_cyk_filebeat_config_output.kafka_cyk215"
codec => "json"
}
#-----全量---------------
jdbc {
type => "all"
jdbc_driver_library => "/opt/mysql-connector-java-5.1.48.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://192.18.2.11:3306/jh_uum2"
jdbc_user => "root"
jdbc_password => "root"
schedule => "* * * * *"
statement => "select * from tbl_sys_login_log"
clean_run => false
jdbc_paging_enabled => "true"
jdbc_page_size => "100"
jdbc_default_timezone =>"Asia/Shanghai"
}
}
filter{
if [type] == "all"{
mutate {
remove_field => "@version"
}
json {
source => "message"
target => "all"
remove_field => ["message"]
}
}
}
output{
if "cas" in [tags]{
elasticsearch {
hosts => ["192.168.9.168:9200"]
index => "cas-%{+YYYY.MM.dd}"
}
}
if "uum" in [tags]{
elasticsearch {
hosts => ["192.168.9.168:9200"]
index => "uum-%{+YYYY.MM.dd}"
}
}
if [type] == "all"{
elasticsearch {
hosts => ["192.168.9.168:9200"]
index => "uumdb"
#将"_id"的值设为mysql的id列
document_id => "%{id}"
#document_type => "base"
}
}
}
- 验证数据
ElasticSearch的9200和9300端口的区别
9200用于外部通讯，基于http协议，程序与es的通信使用9200端口。
9300jar之间就是通过tcp协议通信，遵循tcp协议，es集群中的节点之间也通过9300端口进行通信。
- ES配置文件中
# 设置节点间交互的tcp端口 和 http端口不能一致
transport.tcp.port: 9300
# 设置节点间交互的tcp端口 和 http端口不能一致
transport.tcp.port: 自定义
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
# 设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点
#
discovery.zen.ping.unicast.hosts: [根据实际情况添加格式 "ip:对外暴漏的端口","ip:对外暴漏的端口"]
discovery.zen.ping_timeout: 3s
#
# 设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点(防止脑裂)
discovery.zen.minimum_master_nodes: 2
3721
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
