Kubernetes K8S之Helm部署EFK日志分析系统

最新推荐文章于 2022-08-26 19:28:44 发布

LightZhang666

最新推荐文章于 2022-08-26 19:28:44 发布

阅读量2.3k

点赞数 1

分类专栏： kubernetes ci/cd devops 文章标签： kubernetes k8s ci/cd devops docker

本文链接：https://blog.csdn.net/woshizhangliang999/article/details/112792846

版权

devops 同时被 3 个专栏收录

40 篇文章 4 订阅

订阅专栏

kubernetes

30 篇文章 33 订阅

订阅专栏

ci/cd

30 篇文章 3 订阅

订阅专栏

Kubernetes K8S之Helm部署ELK日志分析系统；由于Logstash比较消耗资源，因此本次我们使用Fluentd实现日志收集（EFK）。

主机配置规划

服务器名称(hostname)	系统版本	配置	内网IP	外网IP(模拟)
k8s-master	CentOS7.7	2C/4G/20G	172.16.1.110	10.0.0.110
k8s-node01	CentOS7.7	2C/4G/20G	172.16.1.111	10.0.0.111
k8s-node02	CentOS7.7	2C/4G/20G	172.16.1.112	10.0.0.112

备注：由于EFK部署比较消耗内存；所以每台机器的内存最好大于等于4G。

ELK概述

ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。

Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等接口，提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。

Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括 syslog、消息传递（例如 RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。

Kibana是一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据，不仅允许用户创建他们自己数据定制仪表板的视图，还允许他们以特殊的方式查询和过滤数据。

由于Logstash比较消耗资源，因此本次我们使用Fluentd实现日志收集（EFK）。

EFK镜像下载

由于镜像都在国外，因此我们在国内下载镜像，然后tag为对应的镜像名称。执行如下脚本【集群所有机器都执行】：

[root@k8s-node02 software]# vim download_efk_image.sh 
#!/bin/sh

##### 在 master 节点和 worker 节点都要执行 【所有机器执行】

# 加载环境变量
. /etc/profile
. /etc/bashrc

# 变量设置
elasticsearch_iamge="elasticsearch-oss:6.7.0"
busybox_image="busybox:latest"
bats_image="bats:0.4.0"
fluentd_image="fluentd-elasticsearch:v2.3.2"
kibana_image="kibana-oss:6.7.0"

# 集群所有机器执行
# elasticsearch镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge}
docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
# busybox镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image}
docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
# bats镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image}
docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
# fluentd-elasticsearch镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image}
docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
# kibana-oss镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image}
docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}

Elasticsearch部署

本次部署EFK，创建一个efk名称空间。

chart下载与配置修改

# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 创建 efk 名称空间
[root@k8s-master efk]# kubectl create namespace efk
[root@k8s-master efk]# 
# ES版本查看，本次我们部署chart 1.30.0版本，ES 6.7.0版本
[root@k8s-master efk]# helm search stable/elasticsearch -l
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/elasticsearch         	1.32.5       	6.8.6      	DEPRECATED Flexible and powerful open source, distributed...
stable/elasticsearch         	1.32.4       	6.8.6      	Flexible and powerful open source, distributed real-time ...
stable/elasticsearch         	1.32.3       	6.8.6      	Flexible and powerful open source, distributed real-time ...
………………
[root@k8s-master efk]# 
[root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0
[root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz
# 修改配置文件1
[root@k8s-master efk]# vim elasticsearch/values.yaml
initImage:
  repository: "busybox"
  tag: "latest"
  pullPolicy: "IfNotPresent"  # 从Always 改为IfNotPresent
………………
client:
  name: client
  replicas: 1  # 从2改为1，因为是在自己PC机操作的，内存有限
  serviceType: ClusterIP
………………
master:
  name: master
  exposeHttp: false
  replicas: 3   # 不要修改
  heapSize: "512m"
  persistence:
    enabled: false  # 没有多余的PVC，因此从true改为false
    accessMode: ReadWriteOnce
………………
data:
  name: data
  exposeHttp: false
  replicas: 1  # 从2改为1，因为是在自己PC机操作的，内存有限
  heapSize: "1024m"  # 从1536m改为1024m，因为是在自己PC机操作的，内存有限
  persistence:
    enabled: false  # 没有多余的PVC，因此从true改为false
    accessMode: ReadWriteOnce
[root@k8s-master efk]# 
# 修改配置文件2
[root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml
apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
kind: Deployment
………………
spec:
  replicas: {{ .Values.client.replicas }}
  # 添加信息 如下
  selector:
    matchLabels:
      app: {{ template "elasticsearch.name" . }}
      component: "{{ .Values.client.name }}"
      release: {{ .Release.Name }}
  # 添加信息 如上
[root@k8s-master efk]# 
# 修改配置文件3
[root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml
apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
kind: StatefulSet
………………
spec:
  serviceName: {{ template "elasticsearch.data.fullname" . }}
  replicas: {{ .Values.data.replicas }}
  # 添加信息 如下
  selector:
    matchLabels:
      app: {{ template "elasticsearch.name" . }}
      component: "{{ .Values.data.name }}"
      release: {{ .Release.Name }}
      role: data
  # 添加信息 如上
[root@k8s-master efk]# 
# 修改配置文件4
[root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml 
apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
kind: StatefulSet
………………
spec:
  serviceName: {{ template "elasticsearch.master.fullname" . }}
  replicas: {{ .Values.master.replicas }}
  # 添加信息 如下
  selector:
    matchLabels:
      app: {{ template "elasticsearch.name" . }}
      component: "{{ .Values.master.name }}"
      release: {{ .Release.Name }}
      role: master
  # 添加信息 如上
[root@k8s-master efk]#

Elasticsearch部署

步骤如下：

# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署ES
[root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/
[root@k8s-master efk]# 
# 状态查看
[root@k8s-master ~]# helm list
NAME          	REVISION	UPDATED                 	STATUS  	CHART                	APP VERSION	NAMESPACE  
es01          	1       	Sat Jul 25 12:18:50 2020	DEPLOYED	elasticsearch-1.30.0 	6.7.0      	efk 
[root@k8s-master efk]# 
# 等待一会儿后【估计几分钟】，查看pod状态信息如下
[root@k8s-master ~]# kubectl get deploy -n efk
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
es01-elasticsearch-client   1/1     1            1           6m13s
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get pod -n efk
NAME                                         READY   STATUS    RESTARTS   AGE
es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          6m21s
es01-elasticsearch-data-0                    1/1     Running   0          6m21s
es01-elasticsearch-master-0                  1/1     Running   0          6m21s
es01-elasticsearch-master-1                  1/1     Running   0          5m30s
es01-elasticsearch-master-2                  1/1     Running   0          5m3s
[root@k8s-master efk]# 
[root@k8s-master efk]# kubectl get svc -n efk
NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
es01-elasticsearch-client      ClusterIP   10.100.237.152  <none>        9200/TCP   6m34s
es01-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP   6m42s
[root@k8s-master efk]# 
[root@k8s-master efk]# kubectl get sts -n efk
NAME                        READY   AGE
es01-elasticsearch-data     1/1     7m4s
es01-elasticsearch-master   3/3     7m4s
[root@k8s-master efk]#

Elasticsearch访问

其中IP来源于ES的svc。

[root@k8s-master ~]# curl 10.100.237.152:9200/
{
  "name" : "es01-elasticsearch-client-646f8f866d-rt2wp",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@k8s-master ~]# 
[root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",   # 可见状态正常
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

至此，elasticsearch部署完毕

elasticsearch-client域名获取

根据es01-elasticsearch-client的svc信息获取到es01-elasticsearch-client的域名；用于后面的fluentd 和kibana。

启动一个pod

[root@k8s-master test]# pwd
/root/k8s_practice/test
[root@k8s-master test]# cat myapp_demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp-demo
  namespace: default
  labels:
    k8s-app: myapp
spec:
  containers:
  - name: myapp
    image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: httpd
      containerPort: 80
      protocol: TCP
[root@k8s-master test]# 
[root@k8s-master test]# kubectl apply -f myapp_demo.yaml
pod/myapp-demo created
[root@k8s-master test]# 
[root@k8s-master test]# kubectl get pod -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
myapp-demo   1/1     Running   0          6s    10.244.2.84   k8s-node02   <none>           <none>

进入pod并得到elasticsearch-client域名信息

# 进入一个pod容器
[root@k8s-master test]# kubectl exec -it myapp-demo sh 
##### 格式 nslookup svc-cluster-ip
/ # nslookup 10.100.237.152
nslookup: can't resolve '(null)': Name does not resolve

Name:      10.100.237.152
Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local
/ # 
/ # 
### 通过ping判断域名是否通畅
/ # ping es01-elasticsearch-client.efk.svc.cluster.local
PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes
64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms
64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms
64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms

由上可得，Service中es01-elasticsearch-client的域名为：es01-elasticsearch-client.efk.svc.cluster.local

Service的域名格式为：$(service name).$(namespace).svc.cluster.local，其中 cluster.local 指定的集群的域名

Fluentd部署

chart下载与配置修改

[root@k8s-master efk]# pwd
/root/k8s_practice/efk
[root@k8s-master efk]#
# fluentd版本信息查看
[root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l
NAME                        	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/fluentd-elasticsearch	2.0.7        	2.3.2      	DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/fluentd-elasticsearch	2.0.6        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch	2.0.5        	2.3.2      	A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
………………
# 获取fluentd-elasticsearch 并解压
[root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7
[root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz
# 配置修改
[root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml
### 为什么使用域名而不是IP，因此每次重启ES的svc，对应IP都会改变。而域名是不变的
elasticsearch:
  host: 'es01-elasticsearch-client.efk.svc.cluster.local'   # 修改处，域名获取参见上文
  port: 9200
  scheme: 'http'
[root@k8s-master efk]#

fluentd-elasticsearch部署

步骤如下：

################ 部署fluentd-elasticsearch
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署fluentd-elasticsearch
[root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch
[root@k8s-master efk]# 
# 状态查看
[root@k8s-master efk]#  helm list
NAME          	REVISION	UPDATED                 	STATUS  	CHART                      	APP VERSION	NAMESPACE  
es01          	1       	Sat Jul 25 12:18:50 2020	DEPLOYED	elasticsearch-1.30.0       	6.7.0      	efk        
fluentd-es01  	1       	Sat Jul 25 12:36:01 2020	DEPLOYED	fluentd-elasticsearch-2.0.7	2.3.2      	efk
[root@k8s-master efk]# 
#查看pod状态信息如下
[root@k8s-master efk]# kubectl get ds -n efk
NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd-es01-fluentd-elasticsearch   2         2         2       2            2           <none>          113s
[root@k8s-master efk]# 
[root@k8s-master efk]# kubectl get pod -n efk -o wide
NAME                                         READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          17m   10.244.2.57    k8s-node02   <none>           <none>
es01-elasticsearch-data-0                    1/1     Running   0          17m   10.244.2.58    k8s-node02   <none>           <none>
es01-elasticsearch-master-0                  1/1     Running   0          17m   10.244.4.241   k8s-node01   <none>           <none>
es01-elasticsearch-master-1                  1/1     Running   0          17m   10.244.2.59    k8s-node02   <none>           <none>
es01-elasticsearch-master-2                  1/1     Running   0          16m   10.244.4.242   k8s-node01   <none>           <none>
fluentd-es01-fluentd-elasticsearch-qnmf9     1/1     Running   0          43s   10.244.4.243   k8s-node01   <none>           <none>
fluentd-es01-fluentd-elasticsearch-xmw5f     1/1     Running   0          43s   10.244.2.60    k8s-node02   <none>           <none>

Kibana部署

kibana的主版本和大版本必须和elasticsearch（ES）一致，小版本可以不同；但两者版本最好一致，这样可以避免一些因版本不同导致的意外情况。

由于elasticsearch（ES）使用的是6.7.0，因此kibana我们也是用该版本。

chart下载与配置修改

[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 所有版本查看
[root@k8s-master efk]# helm search stable/kibana -l
NAME         	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/kibana	3.2.7        	6.7.0      	Kibana is an open source data visualization plugin for El...
stable/kibana	3.2.6        	6.7.0      	Kibana is an open source data visualization plugin for El...
………………
# 获取kibana，并解压缩
[root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7 
[root@k8s-master efk]# tar xf kibana-3.2.7.tgz
# 配置修改1
[root@k8s-master efk]# vim kibana/values.yaml
### 为什么使用域名而不是IP，因此每次重启ES的svc，对应IP都会改变。而域名是不变的
files:
  kibana.yml:
    ## Default Kibana configuration from kibana-docker.
    server.name: kibana
    server.host: "0"
    elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200   # 修改处，域名获取参见上文
………………
service:
  type: NodePort  # 修改内容  从ClusterIP改为NodePort
  externalPort: 443
  internalPort: 5601
  nodePort: 30601  # 添加处，Service端口范围：30000-32767
[root@k8s-master efk]# 
# 配置修改2
[root@k8s-master efk]# vim kibana/templates/deployment.yaml 
apiVersion: apps/v1   # 从 apps/v1beta1 改为 apps/v1
kind: Deployment
metadata:
………………
spec:
  replicas: {{ .Values.replicaCount }}
  revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
  # 添加信息 如下
  selector:
    matchLabels:
      app: {{ template "kibana.name" . }}
      release: "{{ .Release.Name }}"
  # 添加信息 如上

kibana部署

步骤如下：

################ 部署kibana-oss
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署kibana-oss
[root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana
[root@k8s-master efk]#
# 状态查看
[root@k8s-master efk]# helm list
NAME          	REVISION	UPDATED                 	STATUS  	CHART                      	APP VERSION	NAMESPACE  
es01          	1       	Sat Jul 25 12:18:50 2020	DEPLOYED	elasticsearch-1.30.0       	6.7.0      	efk        
fluentd-es01  	1       	Sat Jul 25 12:36:01 2020	DEPLOYED	fluentd-elasticsearch-2.0.7	2.3.2      	efk        
kibana01      	1       	Sat Jul 25 12:38:18 2020	DEPLOYED	kibana-3.2.7               	6.7.0      	efk
[root@k8s-master efk]# 
#查看pod状态信息如下
[root@k8s-master efk]# kubectl get deploy -n efk
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
es01-elasticsearch-client   1/1     1            1           19m
kibana01                    1/1     1            1           27s
[root@k8s-master efk]# 
[root@k8s-master efk]# kubectl get pod -n efk -o wide
NAME                                         READY   STATUS    RESTARTS   AGE     IP             NODE         NOMINATED NODE   READINESS GATES
es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          20m     10.244.2.57    k8s-node02   <none>           <none>
es01-elasticsearch-data-0                    1/1     Running   0          20m     10.244.2.58    k8s-node02   <none>           <none>
es01-elasticsearch-master-0                  1/1     Running   0          20m     10.244.4.241   k8s-node01   <none>           <none>
es01-elasticsearch-master-1                  1/1     Running   0          19m     10.244.2.59    k8s-node02   <none>           <none>
es01-elasticsearch-master-2                  1/1     Running   0          19m     10.244.4.242   k8s-node01   <none>           <none>
fluentd-es01-fluentd-elasticsearch-qnmf9     1/1     Running   0          3m10s   10.244.4.243   k8s-node01   <none>           <none>
fluentd-es01-fluentd-elasticsearch-xmw5f     1/1     Running   0          3m10s   10.244.2.60    k8s-node02   <none>           <none>
kibana01-bc479f8c7-kr2n2                     1/1     Running   0          53s     10.244.4.244   k8s-node01   <none>           <none>
[root@k8s-master efk]# 
# 查看svc信息
[root@k8s-master efk]# kubectl get svc -n efk -o wide
NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE   SELECTOR
es01-elasticsearch-client      ClusterIP   10.100.237.152  <none>        9200/TCP        20m   app=elasticsearch,component=client,release=es01
es01-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP        20m   app=elasticsearch,component=master,release=es01
kibana01                       NodePort    10.101.200.177  <none>        443:30601/TCP   71s   app=kibana,release=kibana01