elasticsearch 、logstash、filebeat、kibana日志执行流程

elasticsearch

1.安装地址:wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-linux-x86_64.tar.gz
2.创建普通账号 注意elasticsearch最后用非root账号运行
2.1创建过程
[root@master ~]# useradd yangyu 
[root@master ~]# passwd yangyu 
Changing password for user yangyu.
3.解压:
tar -zxvf elasticsearch-7.5.1-linux-x86_64.tar.gz
4.修改目录用户和用户组
[root@master ~]# chown -R yangyu:yangyu /home/elasticsearch

5.运行

 $cd /home/elasticsearch

启动。通过 -d 参数,表示后台运行。
$ bin/elasticsearch -d

6.使用普通账号启动elasticsearch可能出现的问题

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
1解决方案之一
解决办法:
#切换到root用户修改
vim /etc/security/limits.conf
# 在最后面追加下面内容
*** hard nofile 65536 
*** soft nofile 65536
***  是启动ES的用户
2解决方案:
在/etc/sysctl.conf文件最后添加一行
vm.max_map_count=262144
执行/sbin/sysctl -p 立即生效
3.解决方案
修改elasticsearch.yml  找到discovery那一块,做如下修改
 cluster.initial_master_nodes: ["node-1","node-2"]修改为:cluster.initial_master_nodes: ["node-1"]

7.

  1. 通过 su 普通用户启动

$ bin/elasticsearch -d

--------------------------------------------------------------------------------------------------------------------------------Logstash

Logstash

  1. 下载 wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.zip
  2. 解压

unzip logstash-7.5.1.zip

cd logstash-7.5.1

  1. 在root账户下启动

使用默认配置文件

执行 nohup bin/logstash -f config/logstash-sample.conf & 命令,后台启动 Logstash 服务。

 

 --------------------------------------------------------------------------------------------------------------------------------

 Beats

Beats 是一个全品类采集器的系列,包含多个:

Filebeat :轻量型日志采集器。

Metricbeat :轻量型指标采集器。

Packetbeat :轻量型网络数据采集器。

Winlogbeat :轻量型 Windows 事件日志采集器。

Auditbeat :轻量型审计日志采集器。

Heartbeat :面向运行状态监测的轻量型采集器。

Functionbeat :面向云端数据的无服务器采集器。

演示日志搜集

  1. 下载
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-linux-x86_64.tar.gz
  2. 解压

解压
$ tar -zxvf filebeat-7.5.1-linux-x86_64.tar.gz
$ cd filebeat-7.5.1-linux-x86_64

  1. 配置默认采集日志地址

Config filebeat.yml 定义inputs和输出地址output 选择的是5044

#=========================== Filebeat inputs
filebeat.inputs:
- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    # - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - /home/logs/spring.log # 配置我们要读取的 Spring Boot 应用的日志

#-------------------------- Elasticsearch output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

测试案例 spring.log

2021-08-04 11:08:21.388  INFO 11220 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.46]
2021-08-04 11:08:21.663  INFO 11220 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-08-04 11:08:21.663  INFO 11220 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1656 ms
2021-08-04 11:08:22.034  INFO 11220 --- [           main] org.apache.spark.SparkContext            : Running Spark version 2.2.0
2021-08-04 11:08:22.353  INFO 11220 --- [           main] org.apache.spark.SparkContext            : Submitted application: sparkTest
2021-08-04 11:08:22.370  INFO 11220 --- [           main] org.apache.spark.SecurityManager         : Changing view acls to: GR-35
2021-08-04 11:08:22.370  INFO 11220 --- [           main] org.apache.spark.SecurityManager         : Changing modify acls to: GR-35
2021-08-04 11:08:22.371  INFO 11220 --- [           main] org.apache.spark.SecurityManager         : Changing view acls groups to: 
2021-08-04 11:08:22.371  INFO 11220 --- [           main] org.apache.spark.SecurityManager         : Changing modify acls groups to: 
2021-08-04 11:08:22.372  INFO 11220 --- [           main] org.apache.spark.SecurityManager         : SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(GR-35); groups with view permissions: Set(); users  with modify permissions: Set(GR-35); groups with modify permissions: Set()
2021-08-04 11:08:23.569  INFO 11220 --- [           main] org.apache.spark.util.Utils              : Successfully started service 'sparkDriver' on port 50792.
2021-08-04 11:08:23.584  INFO 11220 --- [           main] org.apache.spark.SparkEnv                : Registering MapOutputTracker
2021-08-04 11:08:23.599  INFO 11220 --- [           main] org.apache.spark.SparkEnv                : Registering BlockManagerMaster
2021-08-04 11:08:23.602  INFO 11220 --- [           main] o.a.s.s.BlockManagerMasterEndpoint       : Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-08-04 11:08:23.602  INFO 11220 --- [           main] o.a.s.s.BlockManagerMasterEndpoint       : BlockManagerMasterEndpoint up
2021-08-04 11:08:23.608  INFO 11220 --- [           main] o.apache.spark.storage.DiskBlockManager  : Created local directory at C:\Users\GR-35\AppData\Local\Temp\blockmgr-f7ff5fd7-21ad-46b2-9a95-880d9c92c4dc
2021-08-04 11:08:23.634  INFO 11220 --- [           main] o.a.spark.storage.memory.MemoryStore     : MemoryStore started with capacity 891.0 MB
2021-08-04 11:08:23.671  INFO 11220 --- [           main] org.apache.spark.SparkEnv                : Registering OutputCommitCoordinator
2021-08-04 11:08:23.723  INFO 11220 --- [           main] org.spark_project.jetty.util.log         : Logging initialized @5971ms
2021-08-04 11:08:23.771  INFO 11220 --- [           main] org.spark_project.jetty.server.Server    : jetty-9.3.z-SNAPSHOT
2021-08-04 11:08:23.781  INFO 11220 --- [           main] org.spark_project.jetty.server.Server    : Started @6029ms
2021-08-04 11:08:23.798  INFO 11220 --- [           main] o.s.jetty.server.AbstractConnector       : Started ServerConnector@33634f04{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-08-04 11:08:23.798  INFO 11220 --- [           main] org.apache.spark.util.Utils              : Successfully started service 'SparkUI' on port 4040.
2021-08-04 11:08:23.810  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@737d100a{/jobs,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.811  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@3c98781a{/jobs/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.811  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@4601203a{/jobs/job,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.812  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@80bfa9d{/jobs/job/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.812  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@4b039c6d{/stages,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.812  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@507d64aa{/stages/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.812  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@60b34931{/stages/stage,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.813  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@1e0a864d{/stages/stage/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.813  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@3e67f5f2{/stages/pool,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.814  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@4527f70a{/stages/pool/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.814  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@7132a9dc{/storage,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.814  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@2da66a44{/storage/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.815  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@61bfc9bf{/storage/rdd,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.816  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@329bad59{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.818  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@178f268a{/environment,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.819  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@5c723f2d{/environment/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.820  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@2d7a9786{/executors,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.820  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@366d8b97{/executors/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.820  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@7654f833{/executors/threadDump,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.821  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@d7109be{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.827  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@f4a3a8d{/static,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.828  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@1f1ff879{/,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.829  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@24f870ee{/api,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.829  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@1f03fba0{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-08-04 11:08:23.830  INFO 11220 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@3bbf6abe{/stages/stage/kill,null,AVAILABLE,@Spark}

-------------------------------------------------------------------------------------------------------------------------------- 

 Kibana 

1.下载
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-linux-x86_64.tar.gz
2.解压
tar -zxvf kibana-7.5.1-linux-x86_64.tar.gz
cd kibana-7.5.1-linux-x86_64
3.配置
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
kibana.index: ".kibana"
4使用跟elasticsearch 一样用户启动
改变文件属性
 chown -R yangyu:yangyu /home/Kibana
执行 nohup bin/kibana & 命令,后台启动 Kibana 服务。
可以通过 nohup.out 日志,查看启动是否成功。
5 .在使用kibana之前要启动 elasticsearch 、logstash、filebeat
且日志已经写入到配置地址 /home/logs/spring.log
filebeat会自动采集日志信息,通过logstash转换,存储到elasticsearch ,这样才能在kibana显示出来

 

第一步要搜索到filebeat-7.5.1-*,必须第五点执行完成

至此,elasticsearch 、logstash、filebeat安装和具体数据流程已经完毕,下章介绍通过sprinboot-LogBack组件来完成日志生成,使用filebeat进行日志采集

ELK 最简架构

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值