ELK
文章目录
1、 准备环境
1.1 下载安装包
选择
Linux x86_64下载安装包 :
elasticsearch-7.16.3-linux-x86_64.tar.gz
kibana-7.16.3-linux-x86_64.tar.gz
logstash-7.16.3-linux-x86_64.tar.gz
filebeat-7.16.3-linux-x86_64.tar.gz
1.2 rsyslog插件
收集所有日志
[appview@db03 ~]$ ip a | grep 'inet 192.1'
inet 192.168.75.36/24 brd 192.168.75.255 scope global noprefixroute ens33
[appview@db03 ~]$ sudo yum -y install rsyslog
[appview@db03 ~]$ rpm -aq |grep rsyslog
rsyslog-8.24.0-57.el7_9.3.x86_64
[appview@db03 ~]$ ls /etc/rsyslog.d/
listen.conf
[appview@db03 ~]$ cat /etc/rsyslog.d/listen.conf
$SystemLogSocketName /run/systemd/journal/syslog
[root@db03 ~]# vim /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun 514
*.* /var/log/edon.log
[root@db03 ~]# systemctl restart rsyslog.service
[root@db03 ~]# netstat -tunlp | grep 514
udp 0 0 0.0.0.0:514 0.0.0.0:* 2048/rsyslogd
udp6 0 0 :::514 :::* 2048/rsyslogd
[root@db03 ~]# ll /var/log/edon.log
-rw-------. 1 root root 779 7月 16 23:47 /var/log/edon.log
[root@db03 ~]# tailf /var/log/edon.log
# 测试写入日志
[appview@db03 ~]$ logger 'elk测试数据'
1.3 安装jdk
[root@db03 ~]# tar xf jdk-20_linux-x64_bin.tar.gz
[root@db03 ~]# ll
总用量 187080
-rw-------. 1 root root 1421 10月 30 2022 anaconda-ks.cfg
drwxr-xr-x. 9 root root 136 7月 17 00:20 jdk-20.0.1
-rw-r--r--. 1 root root 191562615 7月 17 00:05 jdk-20_linux-x64_bin.tar.gz
[root@db03 ~]# mv jdk-20.0.1 /usr/local/src/
[root@db03 ~]# cd /usr/local/src/jdk-20.0.1/
[root@db03 jdk-20.0.1]# cd bin/
[root@db03 bin]# ./java -version
va version "20.0.1" 2023-04-18
Java(TM) SE Runtime Environment (build 20.0.1+9-29)
Java HotSpot(TM) 64-Bit Server VM (build 20.0.1+9-29, mixed mode, sharing)
79 tar xf jdk-11.0.1_linux-x64_bin.tar.gz
80 ll
81 mv jdk-11.0.1 /usr/local/src/
82 cd /usr/local/src/
83 ll
84 vim /etc/profile
[root@db03 bin]# vim /etc/profile
# JAVA_HOME=/usr/local/src/jdk-20.0.1
JAVA_HOME=/usr/local/src/jdk-11.0.1
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH JAVA_HOME CLASSPATH CATALINA_HOME
1.4 安装filebeat
ip:192.168.75.36
[appview@elk01 ~]$ tar xf elasticsearch-7.16.3-linux-x86_64.tar.gz
[appview@elk01 ~]$ ll
总用量 304032
drwxrwxr-x. 2 appview appview 6 7月 16 18:43 app
drwxr-xr-x. 9 appview appview 155 1月 7 2022 elasticsearch-7.16.3
-rw-rw-r--. 1 appview appview 311327254 7月 16 21:58 elasticsearch-7.16.3-linux-x86_64.tar.gz
[appview@elk01 ~]$ mv elasticsearch-7.16.3 app/
[appview@elk01 ~]$ cd app/elasticsearch-7.16.3/
[appview@elk01 elasticsearch-7.16.3]$ ll
[appview@db03 ~]$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.3-linux-x86_64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 34.2M 100 34.2M 0 0 6872k 0 0:00:05 0:00:05 --:--:-- 8156k
[appview@db03 ~]$ tar xf filebeat-7.16.3-linux-x86_64.tar.gz
[appview@db03 ~]$ ll
总用量 35092
drwxrwxr-x. 5 appview appview 212 7月 17 06:44 filebeat-7.16.3-linux-x86_64
-rw-rw-r--. 1 appview appview 35932836 7月 17 06:43 filebeat-7.16.3-linux-x86_64.tar.gz
[appview@db03 ~]$ cd filebeat-7.16.3-linux-x86_64/
[appview@db03 filebeat-7.16.3-linux-x86_64]$ ll
总用量 128052
-rw-r--r--. 1 appview appview 3778847 1月 7 2022 fields.yml
-rwxr-xr-x. 1 appview appview 125167328 1月 7 2022 filebeat
-rw-r--r--. 1 appview appview 166534 1月 7 2022 filebeat.reference.yml
-rw-------. 1 appview appview 8273 1月 7 2022 filebeat.yml
drwxr-xr-x. 3 appview appview 15 1月 7 2022 kibana
-rw-r--r--. 1 appview appview 13675 1月 7 2022 LICENSE.txt
drwxr-xr-x. 76 appview appview 4096 1月 7 2022 module
drwxr-xr-x. 2 appview appview 4096 1月 7 2022 modules.d
-rw-r--r--. 1 appview appview 1964303 1月 7 2022 NOTICE.txt
-rw-r--r--. 1 appview appview 814 1月 7 2022 README.md
[appview@db03 ~]$ mkdir app
[appview@db03 ~]$ mv filebeat-7.16.3-linux-x86_64 app/
[appview@db03 ~]$ cd app/filebeat-7.16.3-linux-x86_64/
# 修改配置文件
[appview@db03 filebeat-7.16.3-linux-x86_64]$ egrep -v '#|^$' ~/app/filebeat-7.16.3-linux-x86_64/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/edon.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.elasticsearch:
hosts: ["192.168.75.32:9200"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
1.5 安装elasticsearch
1.5.1 安装
ip:192.168.75.32
$ mkdir app
$ tar xf elasticsearch-7.16.3-linux-x86_64.tar.gz -C app/
$ cd app/elasticsearch-7.16.3/config/
$ cp elasticsearch.yml{
,.bak}
# 修改配置文件里面的如下语句,取消注释,删掉, “node-2”
# cluster.initial_master_nodes: [“node-1”, “node-2”]
[appview@elk02 config]$ egrep -v '#|^$' elasticsearch.yml
cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
1.5.2 处理报错
[ERROR][o.e.b.Bootstrap ] [node-1] node validation exception
[2] bootstrap checks failed. You must address the points described in the following [2] lines before starting Elasticsearch.
bootstrap check failure [1] of [2]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
bootstrap check failure [2] of [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]【错误】【o.e.b。[node-1]节点验证异常
[2]引导检查失败。在启动Elasticsearch之前，必须解决以下[2]行中描述的要点。
[2]的引导检查失败[1]:elasticsearch进程的最大文件描述符[4096]太低，增加到至少[65535]
Bootstrap check failure [2] of [2]: Max virtual memory areas vm。Max_map_count[65530]太低，至少增加到[262144]
这个报错意味着在启动 Elasticsearch 之前需要解决两个引导检查（bootstrap checks）失败的问题：
-
max file descriptors（最大文件描述符）过低：Elasticsearch 进程的最大文件描述符数设置过低。建议将其增加至至少 65535。 -
max virtual memory areas vm.max_map_count（最大虚拟内存区域）过低：vm.max_map_count设置过低。建议将其增加至至少 262144。
要解决这些问题，您可以按照以下步骤进行操作：
-
调整最大文件描述符数（max file descriptors）：
-
打开
/etc/security/limits.conf配置文件：sudo vim /etc/security/limits.conf -
在文件末尾添加以下行（如果已经存在，请更新现有的对应行）：
* soft nofile 65536 * hard nofile 65536 * soft nproc 65536 * hard nproc 65536 -
保存并关闭文件。
-
-
调整最大虚拟内存区域（max virtual memory areas）：
-
打开
/etc/sysctl.conf配置文件：sudo vim /etc/sysctl.conf -
在文件末尾添加以下行（如果已经存在，请更新现有的对应行）：
vm.max_map_count=655360 -
保存并关闭文件。
-
-
应用配置更改：
-
运行以下命令以使新的限制和配置生效：
sudo /sbin/sysctl -p -
重新启动 Elasticsearch：
sudo systemctl restart elasticsearch
-
注意改完需要重新连接才能生效
这些步骤将更新系统的限制和配置，以满足 Elasticsearch 启动时的要求。在完成这些更改后，您应该能够成功启动 Elasticsearch。如果问题仍然存在，请确保您正确执行了以上步骤，并检查相关配置文件的语法和格式是否正确。
1.5.3 启动elasticsearch
[appview@elk02 ~]$ cd app/elasticsearch-7.16.3/bin/
[appview@elk02 bin]$ ./elasticsearch -d
[appview@elk02 bin]$ netstat -tunlp | grep 92*
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::9200 :::* LISTEN 2539/java
tcp6 0 0 :::9300 :::* LISTEN 2539/java
[appview@elk02 bin]$ ps -ef |grep el
appview 2539 1 44 14:21 pts/1 00:00:35 /home/appview/app/elasticsearch-7.16.3/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-8404053089288380777 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms1885m -Xmx1885m -XX:MaxDirectMemorySize=988807168 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/home/appview/app/elasticsearch-7.16.3 -Des.path.conf=/home/appview/app/elasticsearch-7.16.3/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /home/appview/app/elasticsearch-7.16.3/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
appview 2566 2539 0 14:21 pts/1 00:00:00 /home/appview/app/elasticsearch-7.16.3/modules/x-pack-ml/platform/linux-x86_64/bin/controller
[appview@elk02 bin]$ curl -X GET 192.168.75.32:9200
{
"name" : "node-1",
"cluster_name" : "my-application",
"cluster_uuid" : "EUFNxa6yT1ezmyC1ML__aA",
"version" : {
"number" : "7.16.3",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "4e6e4eab2297e949ec994e688dad46290d018022",
"build_date" : "2022-01-06T23:43:02.825887787Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
1.6 安装kibana
ip:192.168.75.33
[appview@elk03 ~]$ mkdir app
[appview@elk03 ~]$
[appview@elk03 ~]$
[appview@elk03 ~]$ tar xf kibana-7.16.3-linux-x86_64.tar.gz -C app/
[appview@elk03 ~]$
[appview@elk03 ~]$
[appview@elk03 ~]$ cd app/kibana-7.16.3-linux-x86_64/config/
[appview@elk03 config]$
[appview@elk03 config]$ ls
kibana.yml node.options
[appview@elk03 config]$
[appview@elk03 config]$ cp kibana.yml{
,.bak}
[appview@elk03 config]$ ll
总用量 20
-rw-r--r--. 1 appview appview 5243 1月 7 2022 kibana.yml
-rw-r--r--. 1 appview appview 5243 7月 17 14:45 kibana.yml.bak
-rw-r--r--. 1 appview appview 305 1月 7 2022 node.options
[appview@elk03 config]$ egrep -v '#|^$' kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.75.32:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
[appview@elk03 bin]$ ./kibana
1.7 安装apache
ELK 收集日志流程
(apache日志)
- apache 日志配置文件设置为
json格式- filebeat 读取日志，按
json给 elasticsearch- kibana 读取elasticsearch数据
1.7.1 查看/删除 elasticsearch索引
[appview@elk02 bin]$ curl 127.0.0.1:9200/_cat/indices
green open .kibana_7.16.3_001 6a3_3bUTQga8KJcT9-HaVA 1 0 658 20 2.4mb 2.4mb
green open .geoip_databases EcLwae0JQuCcK0Imp9t6KQ 1 0 42 11 50.2mb 50.2mb
green open .apm-custom-link j2t5z-oPSNuVd02MzN68aQ 1 0 0 0 226b 226b
yellow open filebeat-7.16.3-2023.07.17-000001 MC80Tul_Qm6Pf0IUIO5XbQ 3 1 6891 0 1.4mb 1.4mb
green open .apm-agent-configuration g5fwFyoYTYmXkwrqyt1djA 1 0 0 0 226b 226b
green open .kibana_task_manager_7.16.3_001 DiWAOc7aSl-saFKB7lTcuQ 1 0 17 1275 2.6mb 2.6mb
green open .async-search 3sUUutGXQyq5obOCMHagng 1 0 0 0 249b 249b
green open .tasks nUiOAMr-S-axPRJgUJuW7A 1 0 20 0 52.8kb 52.8kb
# 删除elasticsearech索引
[appview@elk02 bin]$ curl -XDELETE 127.0.0.1:9200/filebeat-7.16.3-2023.07.17-000001
{
"acknowledged":true}[appview@elk02 bin]$
[appview@elk02 bin]$ curl 127.0.0.1:9200/_cat/indices
green open .geoip_databases EcLwae0JQuCcK0Imp9t6KQ 1 0 
最低0.47元/天 解锁文章
323
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
