ELK搭建

最新推荐文章于 2024-06-20 22:05:11 发布

小龙加油！！！

最新推荐文章于 2024-06-20 22:05:11 发布

阅读量246

点赞数

分类专栏： ELK 文章标签： elk

本文链接：https://blog.csdn.net/xiaolong1155/article/details/131819477

版权

ELK 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

# 1.ELK组成

ELK不是一款软件组成，是由三款软件组成的，分别是：ElasticSearch、Logstash、Kibana，其中ES、Logstash均是基于JAVA语言开发的，所以底层需要依赖JAVA JDK工具包。

1.1 ElasticSearch

ElasticSearch是基于JAVA语言开发的开源的、分布式、搜索存储引擎，主要是用于持久化存储日志内容（硬盘上）、同时具备实时的检索、分析、统计功能。类似百度搜索引擎的功能。

1.2 Logstash

Logstash是基于JAVA语言开发的开源的、免费的日志收集工具，主要是用于收集客户端的日志内容（系统、内核、安全、应用日志），同时可以过滤日志内容，最终将日志内容持久化存储到ElasticSearch服务器。每个客户端主机均要安装Logstash日志收集插件。

1.3 Kibana

Kibana是基于Nodejs语言开发的WEB界面程序（UI界面：WEB前端框架），主要是为ElasticSearc、Logstash提供WEB界面操作，方便运维人员、开发人员更加直观的对ELK平台配置、日志分析、日志统计等。

1.4 工作原理

客户端安装Logstash日志收集工具，通过logstash收集客户端应用程序的日志内容，将所有的日志过滤出来，并且存入Elasticsearch 搜索引擎里，然后通过Kibana UI在WEB前端展示给用户，用户需要可以进行查看指定的ES引擎中日志内容。

2.单机部署ES

获取elasticsearch-6.8.8.rpm 包

链接：https://pan.baidu.com/s/1nPvM8hYswipkduN-56DGxg?pwd=d2a6 
提取码：d2a6

2.1 部署jdk

[root@es-0001 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
[root@es-0001 ~]#

2.2 部署ES

[root@es-0001 ELK]# ls
elasticsearch-6.8.8.rpm  filebeat-6.8.8-x86_64.rpm  kibana-6.8.8-x86_64.rpm  logstash-6.8.8.rpm  metricbeat-6.8.8-x86_64.rpm
[root@es-0001 ELK]# yum install elasticsearch-6.8.8.rpm -y

2.3 修改配置文件

[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
55:  network.host: 0.0.0.0

2.4 按照提示启动

[root@es-0001 ELK]# vim /etc/elasticsearch/elasticsearch.yml 
[root@es-0001 ELK]# systemctl daemon-reload
[root@es-0001 ELK]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@es-0001 ELK]# systemctl start elasticsearch.service

[root@es-0001 ELK]# netstat -tnlp |grep 9*00
tcp6       0      0 :::9200                 :::*                    LISTEN      12080/java          
tcp6       0      0 :::9300                 :::*                    LISTEN      12080/java          
[root@es-0003 ELK]#

[root@es-0001 ELK]# curl http://127.0.0.1:9200/
{
  "name" : "TDgvq6G",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "51WkvR2YTA6I4jli8OG0qw",
  "version" : {
    "number" : "6.8.8",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "2f4c224",
    "build_date" : "2020-03-18T23:22:18.622755Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.2",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@es-0003 ELK]#

3.集群安装

3.1 配置hosts文件

cat >>/etc/hosts <<EOF
192.168.199.50    es-0001
192.168.199.51    es-0002
192.168.199.52    es-0003
EOF

3.2 在每台机器安装ES并修改配置文件

[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
17:  cluster.name: my-es     #本集群名称
23:  node.name: es-0001 # 本机主机名
55:  network.host: 0.0.0.0
68:  discovery.zen.ping.unicast.hosts: ["es-0001", "es-0002"]  #配置高可用

重启ES

3.3 查看集群信息

[root@es-0001 ~]#  curl http://127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "my-es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

误点：一定要关闭防火墙和selinux

4.ES基本操作

4.1 集群状态查询

# 查询支持的关键字
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/
# 查具体的信息
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master
# 显示详细信息 ?v
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master?v
# 显示帮助信息 ?help
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master?help

4.2 创建索引

[root@es-0001 ~]# curl -XPUT -H "Content-Type: application/json" http://127.0.0.1:9200/zoey -d '{
    "settings":{
       "index":{
          "number_of_shards": 3, 
          "number_of_replicas": 1
       }
    }
}'

4.3 增加数据

[root@es-0001 ~]#  curl -XPUT -H "Content-Type: application/json" \
                    http://127.0.0.1:9200/zoey/teacher/1 -d '{
                      "职业": "诗人",
                      "名字": "李白",
                      "称号": "诗仙",
                      "年代": "唐"
                  }' 





[root@es-0001 ~]#

{"_index":"zoey","_type":"teacher","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}

4.4 查询数据

[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/zoey/teacher/_search?pretty
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "zoey",
        "_type" : "teacher",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "职业" : "诗人",
          "名字" : "李白",
          "称号" : "诗仙",
          "年代" : "唐"
        }
      }
    ]
  }
}
[root@es-0001 ~]#

[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/zoey/teacher/1?pretty
{
  "_index" : "zoey",
  "_type" : "teacher",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "职业" : "诗人",
    "名字" : "李白",
    "称号" : "诗仙",
    "年代" : "唐"
  }
}
[root@es-0001 ~]#

4.5 修改数据

[root@es-0001 ~]# curl -XPOST -H "Content-Type: application/json" \
   http://127.0.0.1:9200/zoey/teacher/1/_update -d '{ 
   "doc": {"年代":"公元701"}
 }'



{"_index":"zoey","_type":"teacher","_id":"1","_version":2,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":1}[root@es-0001 ~]#

curl -XGET http://127.0.0.1:9200/zoey/teacher/1?pretty
{
  "_index" : "zoey",
  "_type" : "teacher",
  "_id" : "1",
  "_version" : 2,
  "_seq_no" : 1,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "职业" : "诗人",
    "名字" : "李白",
    "称号" : "诗仙",
    "年代" : "公元701"
  }
}

4.6 删除数据

删除一条

[root@es-0001 ~]# curl -XDELETE http://127.0.0.1:9200/zoey/teacher/1
{"_index":"zoey","_type":"teacher","_id":"1","_version":3,"result":"deleted","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2,"_primary_term":1}[root@es-0001 ~]#

删除索引
[root@es-0001 ~]#  curl -XDELETE http://127.0.0.1:9200/zoey

4.7 导入数据

curl -XPOST -H "Content-Type: application/json" http://192.168.199.50:9200/_bulk --data-binary @logs.jsonl

5.安装kibana

yum install kibana-6.8.8-x86_64.rpm

5.1 修改配置文件

vim /etc/kibana/kibana.yml
02  server.port: 5601
07  server.host: "0.0.0.0"
28  elasticsearch.hosts: ["http://es-0002:9200", "http://es-0001:9200"]
113 i18n.locale: "zh-CN"

5.2 启动kibana

[root@es-0003 ELK]# systemctl enable --now kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@es-0003 ELK]# systemctl status kibana.service

5.3 web浏览器访问

5.4 测试

导入数据到es，通过kibana展现
测试数据

curl -XPOST -H "Content-Type: application/json" http://192.168.199.50:9200/_bulk --data-binary @logs.jsonl

因为测试的数据是过去十五分钟，所以没有匹配上，只需要将时间修改

在之前的基础上在继续分析处使用的操作系统

6.logstash

6.1 安装logstash

基于java 需要安装jdk

[root@es-0002 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
[root@es-0002 ~]#

 yum install  logstash-6.8.8.rpm -y

6.2 添加配置文件

logstash没有配置文件，必须自己写

[root@zoey ELK]# cd /etc/logstash/
[root@zoey logstash]# ls
conf.d       log4j2.properties     logstash.yml   startup.options
jvm.options  logstash-sample.conf  pipelines.yml
[root@zoey logstash]#

6.2.1 基础配置

 ln -s /etc/logstash /usr/share/logstash/config

添加配置文件

vim /etc/logstash/conf.d/my.conf

input { 
  stdin {}
}

filter{ }

output{ 
  stdout{}
}

6.2.2 启动logstash

[root@zoey logstash]# pwd
/usr/share/logstash
[root@zoey logstash]# ./bin/logstash

随便输入的转换为json

6.3 插件与调试格式

使用json格式字符串测试 {“a”:“1”, “b”:“2”, “c”:“3”}
直接测试，他没有识别处json

 {"a":"1", "b":"2", "c":"3"}
{
          "host" => "zoey",
       "message" => " {\"a\":\"1\", \"b\":\"2\", \"c\":\"3\"}",
    "@timestamp" => 2023-07-19T13:49:57.512Z,
      "@version" => "1"
}

在配置文件修改

input { 
  stdin { codec => "json" }
}

filter{ }

output{ 
  stdout{ codec => "rubydebug" }
}

重启

[root@zoey bin]# /usr/share/logstash/bin/logstash


{"a":"1", "b":"2", "c":"3"}
{
      "@version" => "1",
             "b" => "2",
             "c" => "3",
          "host" => "zoey",
    "@timestamp" => 2023-07-19T13:59:00.201Z,
             "a" => "1"
}

6.4 file插件

[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input {
  file {
    path => ["/tmp/c.log"]
    type => "test"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb"
  }
}
filter{ }
output{ 
  stdout{ codec => "rubydebug" }
}