EFK 收集 K8S 容器日志
日志在生产环境中非常重要，可以帮助我们排查问题，分析业务问题，分析用户行为等。
对于单机版的集群式应用来说，如果遇到故障我们可以登录上服务器上通过查看日志文件的方式进行查看日志，但对多规模的分布式多节点应用来说，日志分散在多个节点，这显然通过登录上每个节点服务器通过命令查看日志不可取的，就需要统一的日志管理平台收集各各节点上的日志集中管理系统。ELK stack 就是一款日志收集、处理、搜索、分析的成熟的管理系统。
一、ELK简介：
ELK是三个开源软件的缩写，分别为：Elasticsearch 、 Logstash以及Kibana , 它们都是开源软件。不过现在还新增了一个Beats，它是一个轻量级的日志收集处理工具(Agent)，Beats占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具，目前由于原本的ELK Stack成员中加入了 Beats 工具所以已改名为Elastic Stack。
Elasticsearch 主要负责日志存储，索引，分片，搜索等
Logstash 主要负责日志的收集，日志处理，是一个重量级的组件。随后就有Fluentd出现，相比它更易用、资源消耗更少、性能更高，在数据处理上更高效可靠，受到企业欢迎。
F：filebeat
F：Fluentd 日志收集代理工具
二、监控方案
日志管理如果手动处理，可以使用kubectl log -n 名称空间 pod名 (kubectl log -f -n 名称空间 pod名)动态查看日志。但是在集群情况下，由于节点数量多，环境复杂，pod的生命周期等特性，是手动查看日志不现实，而使用 elkstack收集日志我们有两种方案：
| 方案 | 方式 |
|---|---|
| 方案一 | 每个容器中集成日志收集工具 |
| 方案二 | 每个pod中运行两个容器，一个业务主容器，一个日志收集容器 |
| 方案三 | 直接从k8s 宿主机中获取容器日志 【官方推荐】本次测试方案 |
日志收集工具方案:
- Logstash 占内存不推荐使用
- filebeat 轻量15M左右
- fluentd 每台节点运行一个进程，本次测试方案
三、监控原理
k8s集群中的pod产生的日志（stdout/stderr）都存放在对应node的/var/lib/docker/containers/下，使用fluentd的pod，以deamonset形式运行在各个节点之上收集所有pod的日志
四、安装方式
k8s是一个 容器的编排平台，而在k8s之上我们运行的是pod。那么EFK这套环境也运行在k8s上面。elasticsearch是日志收集存储的，在k8s之上我们要使用存储卷来保证数据的稳定性，为了保证eslaticsearch的稳定性，我们采用分角色集群(mast/data/client)。fluentd 需要运行在每台node节点(包括master节点)之上，且每个节点只能运行一个服务，我们采用deamonset控制器控制pod。kibana只是eslasticsearch的web客户端，小规模的集群我们使用deployment运行一个副本即可。运行pod我们需要yml格式的资源清单文件，为了方便起见我们使用helm进行安装。
五、安装es
5.1 搜索helm stable仓库中的elasticsearch 版本
helm search xxx
[root@hw-apptest01-11-172 ~]# helm search stable/elasticsearch -l
NAME CHART VERSION APP VERSION DESCRIPTION
stable/elasticsearch 1.29.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.28.5 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.28.4 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.28.2 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.28.1 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.28.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.27.3 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.27.2 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.26.2 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.26.1 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.26.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.25.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.24.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.23.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.22.0 6.7.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.21.3 6.6.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.21.2 6.6.1 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.21.1 6.6.1 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.21.0 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.20.0 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.19.1 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.19.0 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.18.1 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.18.0 6.6.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.17.2 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.17.1 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.17.0 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.16.0 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.5 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.4 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.3 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.2 6.5.4 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.1 6.5.3 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.15.0 6.5.1 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.14.3 6.5.1 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.14.2 6.5.1 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.14.1 6.5.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.14.0 6.5.0 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.13.3 6.4.3 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.13.2 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.13.1 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.13.0 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.12.0 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.11.1 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch 1.11.0 6.4.2 Flexible and powerful open source, distributed real-time ...
stable/elasticsearch-curator 1.5.0 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.4.0 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.3.2 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.3.1 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.2.1 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.2.0 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.1.0 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.0.1 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-curator 1.0.0 5.5.4 A Helm chart for Elasticsearch Curator
stable/elasticsearch-exporter 1.4.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.4.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.3.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.3.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.2.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.1.3 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.1.2 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.1.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.1.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 1.0.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.4.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.4.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.3.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.2.2 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.2.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.2.0 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.1.4 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.1.3 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.1.2 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.1.1 1.0.2 Elasticsearch stats exporter for Prometheus
stable/elasticsearch-exporter 0.1.0 1.0.2 Elasticsearch stats exporter for Prometheus
本次我们使用较新的版本1.28.5，使用的elasticsearch版本为6.7
5.2 获取 1.28.5 elasticsearch
[root@hw-apptest01-11-172 ~]# helm fetch stable/elasticsearch --version=1.28.5
[root@hw-apptest01-11-172 ~]#
[root@hw-apptest01-11-172 ~]# ls
elasticsearch-1.28.5.tgz
[root@hw-apptest01-11-172 ~]# tar -xf elasticsearch-1.28.5.tgz
tar: elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-auth.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-deployment.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-ingress.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/client-svc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/data-statefulset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/job.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-pdb.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-statefulset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/master-svc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/podsecuritypolicy.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/role.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/rolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/tests/test-configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/templates/tests/test.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/.helmignore: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/README.md: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/expose-transport-port-on-service-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/extrainitcontainers-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/plugin-initcontainer-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: elasticsearch/ci/updatestrategy-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
[root@hw-apptest01-11-172 ~]# cd elasticsearch/
[root@hw-apptest01-11-172 elasticsearch]# ls
Chart.yaml ci README.md templates values.yaml
5.3 修改values.yaml文件
[root@hw-apptest01-11-172 elasticsearch]# vim values.yaml
# 修改数据卷，测试期间我们禁用数据卷
[root@hw-apptest01-11-172 elasticsearch]# grep -A1 -Rn persistence values.yaml
170: persistence:
171- enabled: false
--
217: persistence:
218- enabled: false
# 安装需求修改jvm参数，由于是测试，我们测试效果即可，jvm调至最小
[root@hw-apptest01-11-172 elasticsearch]# egrep -Rn heapSize values.yaml
121: heapSize: "256m"
168: heapSize: "256m"
215: heapSize: "256m"
# 安装需求，修改 elasticsearch 集群每个角色的副本数。(测试可以每个角色单节点，但为了测试每个角色的机器我们不改变集群的规模)
[root@hw-apptest01-11-172 elasticsearch]# egrep -B 3 -Rn replicas values.yaml
109-
110-client:
111- name: client
112: replicas: 2
--
164-master:
165- name: master
166- exposeHttp: false
167: replicas: 3
--
211-data:
212- name: data
213- exposeHttp: false
214: replicas: 2
5.4 elasticsearch 集群角色
stable 官方提供的 elasticsearch 集群配置，是将elasticsearch 按角色做集群，在生产环境下，如果不修改elasticsearch节点的角色信息，在高数据量，高并发的场景下集群容易出现脑裂等问题。
默认情况下，elasticsearch集群中每个节点都有成为主节点的资格，也都存储数据，即双重角色。
由两个属性控制：node.master和node.data，默认情况下这两个属性的值都是true，及既是master 又是data
elasticsearch 集群的三个角色:
- master 负责成为主节点，维护整个集群的状态
- data 负责存储数据,后期提供存储和查询服务,分片
- client 海量请求的时候可以进行负载均衡
参考: https://blog.csdn.net/psc0606/article/details/80247662
5.5 elasticsearch 集群安装
下面我们将es集群安装到我们的k8s中，helm install
# efk 是一组组件，负责我们整个k8s集群的日志管理。为了便于后期的管理，我们新建一个efk的名称空间，将efk集群运行于efk名称空间中
[root@hw-apptest01-11-172 elasticsearch]# kubectl create namespace efk
[root@hw-apptest01-11-172 elasticsearch]# kubectl get ns
NAME STATUS AGE
default Active 127d
efk Active 39d
kube-public Active 127d
kube-system Active 127d
# 安装efk 生产release
[root@hw-apptest01-11-172 elasticsearch]# helm install --name=elstest --namespace=efk -f values.yaml stable/elasticsearch --version=1.28.5
NAME: elstest
LAST DEPLOYED: Wed Jun 26 21:53:35 2019
NAMESPACE: efk
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
elstest-elasticsearch 4 0s
elstest-elasticsearch-test 1 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
elstest-elasticsearch-client-5f68dfbddc-9h6ds 0/1 Init:0/1 0 0s
elstest-elasticsearch-client-5f68dfbddc-lqhx2 0/1 Init:0/1 0 0s
elstest-elasticsearch-data-0 0/1 Init:0/2 0 0s
elstest-elasticsearch-master-0 0/1 Init:0/2 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elstest-elasticsearch-client ClusterIP 10.103.83.183 <none> 9200/TCP 0s
elstest-elasticsearch-discovery ClusterIP None <none> 9300/TCP 0s
==> v1/ServiceAccount
NAME SECRETS AGE
elstest-elasticsearch-client 1 0s
elstest-elasticsearch-data 1 0s
elstest-elasticsearch-master 1 0s
==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
elstest-elasticsearch-client 0/2 2 0 0s
==> v1beta1/StatefulSet
NAME READY AGE
elstest-elasticsearch-data 0/2 0s
elstest-elasticsearch-master 0/3 0s
NOTES:
The elasticsearch cluster has been installed.
Elasticsearch can be accessed:
* Within your cluster, at the following DNS name at port 9200:
elstest-elasticsearch-client.efk.svc
* From outside the cluster, run these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace efk -l "app=elasticsearch,component=client,release=elstest" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:9200 to use Elasticsearch"
kubectl port-forward --namespace efk $POD_NAME 9200:9200
5.6 观察 release 的状态，pod的启动过程
pod 的变化过程
Init --> PodInitializing --> Running --> 1/1
[root@hw-apptest01-11-172 elasticsearch]# kubectl get pod -n efk -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds 0/1 Init:0/1 0 20s <none> hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2 0/1 Running 0 20s 10.244.6.113 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 0/1 Init:0/2 0 20s <none> hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 0/1 Init:0/2 0 20s <none> hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 0/1 Init:1/2 0 31s 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 0/1 PodInitializing 0 32s 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds 0/1 PodInitializing 0 52s 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds 0/1 Running 0 53s 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 0/1 Init:1/2 0 73s 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 0/1 PodInitializing 0 74s 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 0/1 Running 0 75s 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 1/1 Running 0 2m7s 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 0/1 Pending 0 0s <none> <none> <none> <none>
elstest-elasticsearch-master-1 0/1 Pending 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 0/1 Init:0/2 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 0/1 Init:1/2 0 8s 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 0/1 PodInitializing 0 9s 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 0/1 Running 0 10s 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2 1/1 Running 0 2m39s 10.244.6.113 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 1/1 Running 0 33s 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 0/1 Pending 0 0s <none> <none> <none> <none>
elstest-elasticsearch-master-2 0/1 Pending 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 0/1 Init:0/2 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 0/1 Running 0 2m41s 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-9h6ds 1/1 Running 0 2m42s 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 1/1 Running 0 2m45s 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 Pending 0 0s <none> <none> <none> <none>
elstest-elasticsearch-data-1 0/1 Pending 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 Init:0/2 0 0s <none> hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 0/1 Init:1/2 0 19s 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 0/1 PodInitializing 0 20s 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 0/1 Running 0 21s 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 Init:0/2 0 33s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 Init:1/2 0 34s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 PodInitializing 0 35s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 0/1 Running 0 36s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 1/1 Running 0 47s 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 1/1 Running 0 56s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
^C[root@hw-apptest01-11-172 elasticsearch]# kubectl get pod -n efk -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds 1/1 Running 0 5m12s 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2 1/1 Running 0 5m12s 10.244.6.113 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 1/1 Running 0 5m12s 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 1/1 Running 0 2m27s 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 1/1 Running 0 5m12s 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 1/1 Running 0 3m5s 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 1/1 Running 0 2m32s 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-4llr2 1/1 Running 3675 33d 10.244.0.21 hw-apptest01-11-172.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-j8sv4 1/1 Running 3636 33d 10.244.6.89 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-xfqzm 1/1 Running 3026 33d 10.244.1.54 hw-apptest02-11-174.7dtest.cn <none> <none>
5.7 测试集群状态
5.7.1 获取集群 service 地址
[root@hw-apptest01-11-172 elasticsearch]# kubectl -n efk get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elstest-elasticsearch-client ClusterIP 10.103.83.183 <none> 9200/TCP 5m41s
elstest-elasticsearch-discovery ClusterIP None <none> 9300/TCP 5m41s
5.7.2 访问 es，查看版本等信息
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200
{
"name" : "elstest-elasticsearch-client-5f68dfbddc-lqhx2",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "iM0f_xRDTHKbWokLhfiiDg",
"version" : {
"number" : "6.7.0",
"build_flavor" : "oss",
"build_type" : "docker",
"build_hash" : "8453f77",
"build_date" : "2019-03-21T15:32:29.844721Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/node?v
{"error":"Incorrect HTTP method for uri [/_cat/node?v] and method [GET], allowed: [POST]","status":405}[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.244.6.115 35 88 5 0.10 0.25 0.18 mi - elstest-elasticsearch-master-2
10.244.6.114 45 88 6 0.10 0.25 0.18 mi - elstest-elasticsearch-master-1
10.244.1.85 35 39 9 0.01 0.22 0.18 di - elstest-elasticsearch-data-0
10.244.1.87 43 39 4 0.01 0.22 0.18 mi * elstest-elasticsearch-master-0
10.244.1.86 35 39 7 0.01 0.22 0.18 i - elstest-elasticsearch-client-5f68dfbddc-9h6ds
10.244.6.116 36 88 3 0.10 0.25 0.18 di - elstest-elasticsearch-data-1
10.244.6.113 37 88 6 0.10 0.25 0.18 i - elstest-elasticsearch-client-5f68dfbddc-lqhx2
5.7.3 查看集群健康状态
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/health
1561557688 14:01:28 elasticsearch green 7 2 0 0 0 0 0 0 - 100.0%
[root@hw-apptest01-11-172 elasticsearch]# curl 10.103.83.183:9200/_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1561557691 14:01:31 elasticsearch green 7 2 0 0 0 0 0 0 - 100.0%
六、安装fluentd
6.1 查询fluentd的可用包
[root@hw-apptest01-11-172 elasticsearch]# helm search fluentd
NAME CHART VERSION APP VERSION DESCRIPTION
kiwigrid/fluentd-elasticsearch 4.0.0 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd 1.10.0 v2.4.0 A Fluentd Elasticsearch Helm chart for Kubernetes.
stable/fluentd-elasticsearch 2.0.7 2.3.2 DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/sumologic-fluentd 1.0.0 2.3.0 Sumologic Log Collector
stable/fluent-bit 2.0.5 1.1.2 Fast and Lightweight Log/Data Forwarder for Linux, BSD an...
[root@hw-apptest01-11-172 elasticsearch]# helm search fluentd-elasticsearch -l
NAME CHART VERSION APP VERSION DESCRIPTION
kiwigrid/fluentd-elasticsearch 4.0.0 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 3.0.2 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 3.0.1 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 3.0.0 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.12.0 2.5.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.11.1 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.11.0 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.10.1 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.10.0 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.9.1 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.9.0 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.8.3 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.8.2 2.5.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.8.1 2.5.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.8.0 2.5.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.7.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.6.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.5.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.4.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.3.3 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.3.2 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.3.1 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.3.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.2.4 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.2.3 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.2.2 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.2.1 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.2.0 2.4.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.1.2 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.1.1 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
kiwigrid/fluentd-elasticsearch 2.1.0 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.7 2.3.2 DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/fluentd-elasticsearch 2.0.6 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.5 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.4 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.3 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.2 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.1 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.0 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.5.0 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.4.2 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.4.1 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.4.0 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.3.0 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.2.0 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.1.1 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.1.0 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.0.3 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.0.2 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.0.1 2.3.1 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 1.0.0 2.2.0 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
我们看到 stable 和 kiwigrid 都有 fluentd-elasticsearch 的可用版本，我们使用稳定的版本stable的2.0.7
6.2 fluentd-elasticsearch 获取安装包
[root@hw-apptest01-11-172 ~]# tar -xf fluentd-elasticsearch-2.0.7.tgz
tar: fluentd-elasticsearch/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/clusterrole.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/clusterrolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/daemonset.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/pod-security-policy.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/role.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/rolebinding.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/service-account.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/templates/service.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/OWNERS: implausibly old time stamp 1970-01-01 08:00:00
tar: fluentd-elasticsearch/README.md: implausibly old time stamp 1970-01-01 08:00:00
[root@hw-apptest01-11-172 ~]# cd fluentd-elasticsearch/
[root@hw-apptest01-11-172 fluentd-elasticsearch]# ls
Chart.yaml OWNERS README.md templates values.yaml
6.3 修改values配置文件
[root@hw-apptest01-11-172 fluentd-elasticsearch]# vim values.yaml
# 修改镜像地址。默认的镜像地址为google的镜像地址，这个地址国内无法访问
[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -B2 -Rn repository values.yaml
1-image:
2: repository: gcr.azk8s.cn/google-containers/fluentd-elasticsearch
3: #repository: gcr.io/google-containers/fluentd-elasticsearch
# 修改 elasticsearch 的地址端口
[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -A2 -Rn elasticsearch: values.yaml
21:elasticsearch:
22- host: 'elstest-elasticsearch-client.efk.svc'
23- port: 9200
# 取消master污点
## 默认情况下，master是有一个污点的，但fluentd是为了收集日志，master的日志也需要手机，我们修改fluentd的容忍度
[root@hw-apptest01-11-172 fluentd-elasticsearch]# grep -A3 -Rn tolerations: values.yaml
81:tolerations:
82- - key: node-role.kubernetes.io/master
83- operator: Exists
84- effect: NoSchedule
6.4 部署 fluentd ，生产release
# 部署 fluentd ，生产release
[root@hw-apptest01-11-172 fluentd-elasticsearch]# helm install name=flutest --namespace=efk -f values.yaml ../fluentd-elasticsearch-2.0.7.tgz
Release "flutest" has been upgraded. Happy Helming!
LAST DEPLOYED: Thu Jun 27 00:06:45 2019
NAMESPACE: efk
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
flutest-fluentd-elasticsearch 33d
==> v1/ClusterRoleBinding
NAME AGE
flutest-fluentd-elasticsearch 33d
==> v1/ConfigMap
NAME DATA AGE
flutest-fluentd-elasticsearch 6 33d
==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
flutest-fluentd-elasticsearch 3 3 1 0 1 <none> 33d
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
flutest-fluentd-elasticsearch-67hfd 1/1 Running 7 90m
flutest-fluentd-elasticsearch-g9m47 0/1 Terminating 12 48m
flutest-fluentd-elasticsearch-qhfh8 0/1 Terminating 14 49m
==> v1/ServiceAccount
NAME SECRETS AGE
flutest-fluentd-elasticsearch 1 33d
NOTES:
1. To verify that Fluentd has started, run:
kubectl --namespace=efk get pods -l "app.kubernetes.io/name=fluentd-elasticsearch,app.kubernetes.io/instance=flutest"
THIS APPLICATION CAPTURES ALL CONSOLE OUTPUT AND FORWARDS IT TO elasticsearch . Anything that might be identifying,
including things like IP addresses, container images, and object names will NOT be anonymized.
[root@hw-apptest01-11-172 fluentd-elasticsearch]# kubectl -n efk get pod -w
NAME READY STATUS RESTARTS AGE
elstest-elasticsearch-client-5f68dfbddc-9h6ds 1/1 Running 0 133m
elstest-elasticsearch-client-5f68dfbddc-lqhx2 1/1 Running 0 133m
elstest-elasticsearch-data-0 1/1 Running 0 133m
elstest-elasticsearch-data-0 1/1 Running 0 133m
elstest-elasticsearch-data-1 1/1 Running 0 130m
elstest-elasticsearch-master-0 1/1 Running 0 133m
elstest-elasticsearch-master-1 1/1 Running 0 131m
elstest-elasticsearch-master-2 1/1 Running 0 130m
flutest-fluentd-elasticsearch-67hfd 1/1 Running 7 91m
flutest-fluentd-elasticsearch-7rph9 0/1 ContainerCreating 0 7s
flutest-fluentd-elasticsearch-rbjrx 0/1 ContainerCreating 0 7s
kibtest-kibana-7c4c7cd9df-whqtz 1/1 Running 0 118m
^[[A
^C[root@hw-apptest01-11-172 fluentd-elasticsearch]# kubectl -n efk get pod -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds 1/1 Running 0 133m 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2 1/1 Running 0 133m 10.244.6.113 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 1/1 Running 0 133m 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 1/1 Running 0 131m 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 1/1 Running 0 133m 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 1/1 Running 0 131m 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 1/1 Running 0 131m 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-7rph9 1/1 Running 0 35s 10.244.6.121 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-rbjrx 1/1 Running 0 35s 10.244.1.92 hw-apptest02-11-174.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-wkj24 0/1 ContainerCreating 0 6s <none> hw-apptest01-11-172.7dtest.cn <none> <none>
kibtest-kibana-7c4c7cd9df-whqtz 1/1 Running 0 118m 10.244.1.88 hw-apptest02-11-174.7dtest.cn <none> <none>
6.5 测试es是否收集到数据
[root@hw-apptest01-11-172 fluentd-elasticsearch]# curl 10.103.83.183:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2019.06.25 bEYYLdDiSMaq-C3iMXsl0g 5 1 5577 0 3.9mb 2mb
green open logstash-2019.05.04 ZiWVSgV_TTKduZa3qmyQqA 5 1 124 0 510.1kb 291.7kb
green open logstash-2019.06.23 V4R9twEfTry-T5vazFAX4Q 5 1 7644 0 5.8mb 3mb
green open logstash-2019.06.11 SdS9nrATTwmZ8E3yi83_nw 5 1 2 0 29.4kb 460b
green open logstash-2019.06.12 aktDQ6IwTnyd78TkEeHuGQ 5 1 17 0 47.7kb 460b
green open logstash-2019.05.30 HYXFwTHPRsCx7V-QJ-FBNw 5 1 164 0 412.1kb 227.1kb
green open logstash-2019.06.06 umcCJRKFS9CnN-zD0eeL6A 5 1 522 0 802.3kb 422kb
green open logstash-2019.06.24 65DW7sVVS5qo17vEmxib6A 5 1 10212 0 6.8mb 3.6mb
green open logstash-2019.06.22 Rg6mACLBQlWjVFkcpmd83A 5 1 1872 0 1.7mb 904.1kb
yellow open logstash-2019.06.26 NLNGeADgT4avG7khGocqfg 5 1 0 0 460b 460b
七、安装kibana
7.1 查询 kibana 的可用包
[root@hw-apptest01-11-172 elasticsearch]# helm search kibana
NAME CHART VERSION APP VERSION DESCRIPTION
elastic/kibana 7.1.1 7.1.1 Kibana
local/kibana 3.2.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 3.2.0 6.7.0 Kibana is an open source data visualization plugin for El...
[root@hw-apptest01-11-172 elasticsearch]# helm search stable/kibana -l
NAME CHART VERSION APP VERSION DESCRIPTION
stable/kibana 3.2.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 3.1.1 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 3.1.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 3.0.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 2.3.1 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 2.3.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 2.2.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 2.1.0 6.7.0 Kibana is an open source data visualization plugin for El...
stable/kibana 2.0.0 6.6.1 Kibana is an open source data visualization plugin for El...
stable/kibana 1.6.0 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.5.2 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.5.1 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.5.0 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.4.1 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.4.0 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.3.0 6.6.0 Kibana is an open source data visualization plugin for El...
stable/kibana 1.2.2 6.5.4 Kibana is an open source data visualization plugin for El...
stable/kibana 1.2.1 6.5.4 Kibana is an open source data visualization plugin for El...
stable/kibana 1.2.0 6.5.4 Kibana is an open source data visualization plugin for El...
stable/kibana 1.1.2 6.5.4 Kibana is an open source data visualization plugin for El...
stable/kibana 1.1.1 6.5.3 Kibana is an open source data visualization plugin for El...
stable/kibana 1.1.0 6.5.3 Kibana is an open source data visualization plugin for El...
stable/kibana 1.0.3 6.5.3 Kibana is an open source data visualization plugin for El...
stable/kibana 1.0.2 6.5.1 Kibana is an open source data visualization plugin for El...
stable/kibana 1.0.1 6.5.1 Kibana is an open source data visualization plugin for El...
stable/kibana 1.0.0 6.5.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.20.0 6.5.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.19.0 6.5.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.18.0 6.4.3 Kibana is an open source data visualization plugin for El...
stable/kibana 0.17.1 6.4.3 Kibana is an open source data visualization plugin for El...
stable/kibana 0.17.0 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.16.4 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.16.3 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.16.2 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.16.1 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.16.0 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.15.0 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.8 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.7 6.4.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.6 6.4.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.5 6.4.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.4 6.4.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.3 6.4.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.2 6.4.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.14.1 6.4.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.13.1 6.4.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.13.0 6.4.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.12.1 6.3.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.12.0 6.3.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.11.0 6.3.2 Kibana is an open source data visualization plugin for El...
stable/kibana 0.10.1 6.3.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.10.0 6.3.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.9.0 6.3.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.8.0 6.3.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.7.0 6.3.1 Kibana is an open source data visualization plugin for El...
stable/kibana 0.6.0 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.5.0 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.4.1 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.4.0 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.3.0 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.2.3 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.2.2 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.2.1 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.2.0 6.0.0 Kibana is an open source data visualization plugin for El...
stable/kibana 0.1.1 5.4.3 Kibana is an open source data visualization plugin for El...
stable/kibana 0.1.0 5.4.3 Kibana is an open source data visualization plugin for El...
7.2 获取 kibana 安装包
[root@hw-apptest01-11-172 ~]# helm fetch stable/kibana --version=3.2.0
[root@hw-apptest01-11-172 ~]# tar -xf kibana-3.2.0.tgz
tar: kibana/Chart.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/NOTES.txt: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/_helpers.tpl: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/configmap-dashboardimport.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/deployment.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/ingress.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/service.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/serviceaccount.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/tests/test-configmap.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/tests/test.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/templates/volume-claim.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/.helmignore: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/OWNERS: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/README.md: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/authproxy-enabled.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/dashboard-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/extra-configmap-mounts.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/ingress-hosts-paths.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/ingress-hosts.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/initcontainers-all-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/initcontainers-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/plugin-install.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/pvc.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/security-context.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/service-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
tar: kibana/ci/url_dashboard-values.yaml: implausibly old time stamp 1970-01-01 08:00:00
[root@hw-apptest01-11-172 ~]# cd kibana/
[root@hw-apptest01-11-172 kibana]# ls
Chart.yaml ci OWNERS README.md templates values.yaml
7.3 修改 values 配置文件
[root@hw-apptest01-11-172 kibana]# vim values.yaml
# 修改 kibana 连接elasticsearch地址
[root@hw-apptest01-11-172 kibana]# grep -Rn -A5 kibana.yml values.yaml
24: kibana.yml:
25- ## Default Kibana configuration from kibana-docker.
26- server.name: kibana
27- server.host: "0"
28- ## For kibana < 6.6, use elasticsearch.url instead
29- elasticsearch.hosts: http://elstest-elasticsearch-client.efk.svc:9200
# 修改 service 的类型为NodePort ，方便我们外网访问
[root@hw-apptest01-11-172 kibana]# grep -Rn -A5 ^service: values.yaml
40:service:
41- type: NodePort
42- # clusterIP: None
43- # portName: kibana-svc
44- externalPort: 443
45- internalPort: 5601
7.4 安装 kibana 生产release
[root@hw-apptest01-11-172 kibana]# helm install --name=kibtest --namespace=efk -f values.yaml stable/kibana --version=3.2.0
NAME: kibtest
LAST DEPLOYED: Wed Jun 26 22:08:35 2019
NAMESPACE: efk
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
kibtest-kibana 1 0s
kibtest-kibana-test 1 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
kibtest-kibana-7c4c7cd9df-whqtz 0/1 ContainerCreating 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kibtest-kibana NodePort 10.103.27.234 <none> 443:30517/TCP 0s
==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
kibtest-kibana 0/1 1 0 0s
NOTES:
To verify that kibtest-kibana has started, run:
kubectl --namespace=efk get pods -l "app=kibana"
Kibana can be accessed:
* From outside the cluster, run these commands in the same shell:
export NODE_PORT=$(kubectl get --namespace efk -o jsonpath="{.spec.ports[0].nodePort}" services kibtest-kibana)
export NODE_IP=$(kubectl get nodes --namespace efk -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
7.5 查看kibana的启动过程
[root@hw-apptest01-11-172 kibana]# kubectl get pod -n efk -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elstest-elasticsearch-client-5f68dfbddc-9h6ds 1/1 Running 0 15m 10.244.1.86 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-client-5f68dfbddc-lqhx2 1/1 Running 0 15m 10.244.6.113 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-data-0 1/1 Running 0 15m 10.244.1.85 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-data-1 1/1 Running 0 12m 10.244.6.116 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-0 1/1 Running 0 15m 10.244.1.87 hw-apptest02-11-174.7dtest.cn <none> <none>
elstest-elasticsearch-master-1 1/1 Running 0 13m 10.244.6.114 hw-apptest03-11-1761.7dtest.cn <none> <none>
elstest-elasticsearch-master-2 1/1 Running 0 12m 10.244.6.115 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-4llr2 1/1 Running 3676 33d 10.244.0.21 hw-apptest01-11-172.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-j8sv4 1/1 Running 3637 33d 10.244.6.89 hw-apptest03-11-1761.7dtest.cn <none> <none>
flutest-fluentd-elasticsearch-xfqzm 1/1 Running 3027 33d 10.244.1.54 hw-apptest02-11-174.7dtest.cn <none> <none>
kibtest-kibana-7c4c7cd9df-whqtz 1/1 Running 0 22s 10.244.1.88 hw-apptest02-11-174.7dtest.cn <none> <none>
7.6 查看 kibana service，访问测试
[root@hw-apptest01-11-172 kibana]# kubectl -n efk get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elstest-elasticsearch-client ClusterIP 10.103.83.183 <none> 9200/TCP 15m
elstest-elasticsearch-discovery ClusterIP None <none> 9300/TCP 15m
flutest-fluentd-elasticsearch ClusterIP 10.96.212.237 <none> 24231/TCP 33d
kibtest-kibana NodePort 10.103.27.234 <none> 443:30517/TCP 41s
八、安装过程中的坑
8.1 helm 仓库地址
问题：
默认情况下的 helm stable版本是google的地址，google的地址再国内无法访问，或时而无法访问。这钟情况下helm仓库使用的第一步就卡死了。
https://kubernetes-charts.storage.googleapis.com/
解决方案：
-
阿里云helm仓库代理
地址:https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
已经好久没有更新过了 -
翻墙
linux 使用vpn 或ss -
内部仓库–git
github官方地址: https://github.com/helm/charts.git -
本地文件–git [详情加helm 进阶文档]
helm serve --address 0.0.0.0:8080 --repo-path /app/dcos/chart-repo
helm repo index /app/dcos/chart-repo
- 第三方代理
代理地址: https://burdenbear.github.io/kube-charts-mirror/
代理工具地址： https://github.com/taowujie/kube-charts-mirror
8.2 访问国外gcr docker镜像
问题:
gcr docker镜像 是google的docker仓库镜像，国内无法访问。无论是后期软件 合适kubeadm安装kubernetes 集群都是一个卡点。
解决方案:
- 方案一 国内加速镜像 https://github.com/anjia0532/gcr.io_mirror
gcr.io/namespace/image_name:image_tag
#eq
gcr.azk8s.cn/namespace/image_name:image_tag
#替换方式---> 微软云国内加速镜像
-
方案二 自己通过 Dockerfile 制作镜像 利用dockerhub生成
（https://blog.csdn.net/weixin_39961559/article/details/80739352
https://blog.csdn.net/qq_27028561/article/details/79064414
https://blog.csdn.net/nklinsirui/article/details/80581286 ） -
方案三 vpn 代理
-
方案四 使用第三方镜像 安全性无法保障
docker search xx
8.3 kibana 无法启动
查看日志:
"message":"child \"elasticsearch\" fails because [\"hosts\" is not allowed]"
官方charts 文件bug，版本es 6.5 版本
url链接:
https://github.com/helm/charts/issues/14062
https://github.com/elastic/kibana/issues/32303
8.4 kibana 不兼容 es
问题:
Kibana server is not ready yet
排查思路
- kibana版本要和elasticsearch版本相同
- elasticsearch 集群状态不正常。
使用kubectl log xxpod名查看日志
8.5 fluentd 无法收集日志到es中
启动正常就是es上面没有日志索引。
原因:
fluentd 的镜像保存在grc docker 仓库中，国内无法访问。起初使用的是第三方镜像，是版本不兼容造成的。
九、优缺点
Kubernetes官方提供了EFK的日志收集解决方案，但是这种方案并不适合所有的业务场景，它本身就有一些局限性，例如：
所有日志都必须是out前台输出，真实业务场景中无法保证所有日志都在前台输出
只能有一个日志输出文件，而真实业务场景中往往有多个日志输出文件
Fluentd并不是常用的日志收集工具，我们更习惯用logstash，现使用filebeat替代
我们已经有自己的ELK集群且有专人维护，没有必要再在kubernetes上做一个日志收集服务
基于以上几个原因，我们决定使用自己的ELK集群。
| 编号 | 方案 | 优点 | 缺点 |
|---|---|---|---|
| 1 | 每个app的镜像中都集成日志收集组件 | 部署方便，kubernetes的yaml文件无须特别配置，可以为每个app自定义日志收集配置 | 强耦合，不方便应用和日志收集组件升级和维护且会导致镜像过大 |
| 2 | 单独创建一个日志收集组件跟app的容器一起运行在同一个pod中 | 低耦合，扩展性强，方便维护和升级 | 需要对kubernetes的yaml文件进行单独配置，略显繁琐 |
| 3 | 将所有的Pod的日志都挂载到宿主机上，每台主机上单独起一个日志收集Pod | 完全解耦，性能最高，管理起来最方便 | 需要统一日志收集规则，目录和输出方式 |
2516
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
