使用 Helm3 部署 Loki

临江仙我亦是行人

已于 2022-06-27 19:39:13 修改

阅读量2.4k

点赞数 1

分类专栏： Loki 文章标签：运维 java linux

于 2022-06-26 18:37:55 首次发布

本文链接：https://blog.csdn.net/weixin_56752399/article/details/125472638

版权

Loki 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

1 为什么选择loki

https://github.com/grafana/loki

Loki 是 Grafana Labs 团队的开源项目，是一个水平可扩展，高可用性，多租户的日志聚合系统。它的设计非常经济高效且易于操作，因为它不会为日志内容设置索引，而是为每个日志流设置一组标签（就像 Prometheus 标签一样），官方的介绍就是：Like Prometheus, but for logs.，类似于 Prometheus 的日志系统

在这里插入图片描述
架构如下

Promtail：Loki 默认客户端，负责采集并上报日志
Distributor：日志写入入口，负责将数据转发到 Ingester
Ingester：日志的写入服务，缓存并写入日志内容和索引到底层存储
Querier：日志读取服务，执行搜索请求

Grafana Loki 主要由 3 部分组成:

Loki：主服务，日志记录引擎，负责存储日志和处理查询

https://github.com/grafana/loki
https://grafana.com/docs/loki/latest/installation/

Promtail：代理服务，负责收集日志并将其发送给 loki

https://grafana.com/docs/loki/latest/clients/promtail/installation/

Grafana：UI 界面，通过 Web 界面来提供数据展示、查询、告警等功能。

2 安装部署

如果没有安装过 Helm3 请安装

# 下载二进制包
$ wget https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz
# 解压二进制包
$ tar zxf helm-v3.5.2-linux-amd64.tar.gz
# 复制可执行文件
$ cp -pr linux-amd64/helm /usr/local/bin/
# 验证安装是否成功
$ helm --help

2.1 部署 Loki

https://grafana.com/docs/loki/latest/installation/simple-scalable-helm/

使用 Helm 将 Loki 部署到 K8s 集群

# 添加 Loki 的 Chart 仓库
helm repo add grafana https://grafana.github.io/helm-charts
# 更新仓库
helm repo update
# 搜索 loki（Helm2：helm search loki）
root@tjdata05:~# helm search repo loki
NAME                        	CHART VERSION	APP VERSION	DESCRIPTION
grafana/loki                	2.12.2       	v2.5.0     	Loki: like Prometheus, but for logs.
grafana/loki-canary         	0.8.1        	2.5.0      	Helm chart for Grafana Loki Canary
grafana/loki-distributed    	0.49.0       	2.5.0      	Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable	1.4.1        	2.5.0      	Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack          	2.6.5        	v2.4.2     	Loki: like Prometheus, but for logs.
grafana/fluent-bit          	2.3.1        	v2.1.0     	Uses fluent-bit Loki go plugin for gathering lo...
grafana/promtail            	6.0.0        	2.5.0      	Promtail is an agent which ships the contents o...
root@tjdata05:~#

grafana/loki 只有单独的 loki 组件
grafana/loki-canary loki 的另外一个组件，主要给 loki 做性能分析
grafana/loki-distributed 分布式（微服务）模式
grafana/loki-stack loki 的工具栈
grafana/fluent-bit 日志收集客户端
grafana/promtail loki 自带的日志收集客户端

获取 loki-stack 的 Chart 包并解压：

# Helm3 
helm pull grafana/loki-stack --untar --version 2.6.5  # 2.6.5 是 CHART VERSION
# 如果是 Helm2 
helm fetch grafana/loki-stack --untar --version 2.6.5

loki-stack 这个 Chart 包里面包含所有的 Loki 相关工具依赖，在安装的时候可以根据需要开启或关闭。默认情况下 loki、promtail 是自动开启的，也可以根据需要选择使用 filebeat 或者 logstash，同样在 Chart 包根目录下自定义安装的 Values 文件：

root@tjdata05:~/loki-stack# cat values-prod.yaml
loki:
  enabled: true
  replicas: 1
  persistence:
    enabled: true
    storageClassName: ceph-rbd
    accessModes:
      - ReadWriteOnce
    size: 80Gi

promtail:
  enabled: true

grafana:
  enabled: true
  service:
    type: ClusterIP
    port: 80
    targetPort: 3000
  persistence:
    enabled: true
    storageClassName: ceph-rbd
    accessModes:
      - ReadWriteOnce
    size: 10Gi
  ingress:
    enabled: true
    path: /
    # pathType is only for k8s >= 1.1=
    pathType: Prefix
    hosts:
      - loki.test.cn
root@tjdata05:~/loki-stack#

然后直接使用上面的 values-prod.yaml文件进行安装即可

root@tjdata05:~/loki-stack# kubectl create ns logging
namespace/logging created
root@tjdata05:~/loki-stack# helm upgrade --install loki -n logging -f values-prod.yaml .
Release "loki" does not exist. Installing it now.
NAME: loki
LAST DEPLOYED: Fri Jun 24 17:47:04 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See http://docs.grafana.org/features/datasources/loki/ for more detail.
root@tjdata05:~/loki-stack#

安装完成后可以查看 Pod 的状态

root@tjdata05:~# kubectl -n logging get pods
NAME                            READY   STATUS    RESTARTS   AGE
loki-0                          1/1     Running   0          4m32s
loki-grafana-7595cff4f5-4szxs   2/2     Running   0          4m31s
loki-promtail-2n28d             1/1     Running   0          4m32s
loki-promtail-46q2l             1/1     Running   0          4m32s
loki-promtail-6fjfm             1/1     Running   0          4m32s
loki-promtail-8fzzp             1/1     Running   0          4m32s
loki-promtail-8ndgj             1/1     Running   0          4m32s
loki-promtail-h7nfx             1/1     Running   0          4m32s
loki-promtail-nn25b             1/1     Running   0          4m32s
loki-promtail-r7nmv             1/1     Running   0          4m32s
loki-promtail-xbxpg             1/1     Running   0          4m32s
root@tjdata05:~#

查看 Service 信息

root@tjdata05:~# kubectl -n logging get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
loki              ClusterIP   10.68.124.157   <none>        3100/TCP   4m9s
loki-grafana      ClusterIP   10.68.25.148    <none>        80/TCP     4m9s
loki-headless     ClusterIP   None            <none>        3100/TCP   4m9s
loki-memberlist   ClusterIP   None            <none>        7946/TCP   4m9s
root@tjdata05:~#

查看 ingress 信息

root@tjdata05:~# kubectl -n logging get ing
NAME           HOSTS              ADDRESS   PORTS   AGE
loki-grafana   loki.test.cn             80      5m6s
root@tjdata05:~#

查看 pvc 信息

root@tjdata05:~# kubectl -n logging get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
loki-grafana     Bound    pvc-4186c1a5-584b-45c1-bc3c-c432c545907e   10Gi       RWO            ceph-rbd       7m40s
storage-loki-0   Bound    pvc-1b6f111c-5292-4e54-b87c-45657d406500   80Gi       RWO            ceph-rbd       2d23h
root@tjdata05:~#

获取 Grafana 的登录密码

kubectl get secret --namespace logging loki-grafana -o jsonpath="{.data.admin-password}" | \
base64 --decode ; echo

# 示例
root@tjdata05:~# kubectl get secret --namespace logging loki-grafana -o jsonpath="{.data.admin-password}" | \
> base64 --decode ; echo
zWWmbfmHkhRBsJONZHiambvgn
root@tjdata05:~#

使用用户名 admin 和上面的获取的密码即可登录 Grafana，由于 Helm Chart 已经为 Grafana 配置好了 Loki 的数据源，所以我们可以直接获取到日志数据了。点击左侧 Explore 菜单，然后就可以筛选 Loki 的日志数据了：

https://loki.test.cn
user:admin
pass:zWWmbfmHkhRBsJONZHiambvgn

在这里插入图片描述

注意：

Loki 默认显示 1000 行
为了方便查看，也可以导入 Dashboard

https://grafana.com/grafana/dashboards/15141

我们使用 Helm 安装的 Promtail 默认已经帮我们做好了配置，已经针对 K8s 做了优化，并且将 Promtail 配置进行了加密：

kubectl -n logging get secret loki-promtail -o yaml
# 解密
echo "" | base64 -d

内容如下

server:
  log_level: info
  http_listen_port: 3101

client:
  url: http://loki:3100/loki/api/v1/push


positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - namespace
        - app
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
        - __meta_kubernetes_pod_container_name
        target_label: __path__

通过 api 查询 loki 中日志

root@tjdata05:~# kubectl -n logging get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
loki              ClusterIP   10.68.124.157   <none>        3100/TCP   88m
loki-grafana      ClusterIP   10.68.25.148    <none>        80/TCP     88m
loki-headless     ClusterIP   None            <none>        3100/TCP   88m
loki-memberlist   ClusterIP   None            <none>        7946/TCP   88m
root@tjdata05:~#

# 打印当前 loki 中所有的标签
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/labels
{"status":"success","data":["__name__","app","component","container","filename","job","namespace","node_name","pod"]}

# 打印标签（node_name）的值
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/label/node_name/values
{"status":"success","data": # 省略

# 打印标签（pod）的值
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/label/pod/values