Loki安装部署并收集K8S日志
loki 官方推荐的最佳实践为采用 DamonSet部署 promtail 的方式,将 node 的 /var/lib/pods目录挂载进容器内部,借助prometheus 的服务发现机制动态的为日志加上标签,无论是资源的占用程度还是部署维护难度都是非常低。这也是主流的云原生日志采集范式。
Loki收集k8s架构图:
安装Loki
采用 helm 方法部署 创建名称空间
kubectl create namespace monitoring
安装Loki 注意:
不做持久化:将 persistence.enabled=true 改为 persistence.enabled=false 即可
时间本地化
service.httpNodePort 指定http端口
service.grpcNodePort 指定grpc端口
serviceMonitor.enabled 让普罗米修斯监控到指标
--set serviceMonitor.labels="prometheus-operated"
k8s版本低无法安装修改版本:kubeVersion: 1.10.0-0
--set ingress.enabled=ture 获取ingress指标
添加仓库:
helm repo add kube-ops https://charts.kube-ops.io
helm repo update
部署 Loki
helm install loki -n monitoring \
--set replicas=1 \
--set promtail.enabled=true \
--set service.type=ClusterIP \
--set persistence.enabled=false \
--set extraVolumeMounts[0].name=localtime \
--set extraVolumeMounts[0].mountPath="/etc/localtime" \
--set extraVolumes[0].name=localtime \
--set extraVolumes[0].hostPath.path="/etc/localtime" \
kube-ops/loki --version 1.6.1
查看pod状态
[root@master ~]# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
loki-0 1/1 Running 1 12h
loki-promtail-99ztv 1/1 Running 0 3m20s
loki-promtail-nb2q4 1/1 Running 0 4m50s
loki-promtail-pzxcl 1/1 Running 0 2m34s
配置Loki为数据源:确认Loki服务端口: Loki服务端口号是80
[root@master ~]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loki ClusterIP 10.233.2.239 <none> 80/TCP,9095/TCP 12h
导入Dashboard
可以看到日志来了
Logging Dashboard via Loki dashboard for Grafana | Grafana Labs
解决Panel plugin not found: grafana-piechart-panel
[root@master ~]# kubectl exec -it grafana-757fcd5f7c-lkxrx -n ops
error: you must specify at least one command for the container
[root@master ~]# kubectl exec -it grafana-757fcd5f7c-lkxrx -n ops bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.0$ grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.2
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.2/download
into: /var/lib/grafana/plugins
?.Installed grafana-piechart-panel successfully
Restart grafana after installing plugins . <service grafana-server restart>
bash-5.0$ service grafana-server restart
bash: service: command not found
bash-5.0$ exit
exit
command terminated with exit code 127
[root@master ~]# kubectl delete pod grafana-757fcd5f7c-lkxrx -n ops
pod "grafana-757fcd5f7c-lkxrx" deleted
最后你要查看日志可以用上面导入的面板,也可以使用之前的源查看