Loki+Grafana(外)采集Kubernetes(K8s)集群(基于containerd)

一、Loki简介

---

1、简介

Loki是一个开源、分布式的日志聚合系统，由Grafana Labs推出。Loki的设计目标是为了高效地处理大规模的日志数据，并具有良好的可扩展性。Loki的最大优点是它具有低资源占用和高效的查询速度。这是因为Loki不需要在处理日志数据时进行索引，而是将数据存储在类似于散列表的内存数据结构中，可以快速地定位和检索日志数据。此外，Loki可以与Prometheus集成，实现更强大的监控功能。

2、组件

• distributor：是负责获取日志数据并将其转发给ingester的无状态组件。Distributor对数据进行预处理，检查其有效性，并确保其来自已配置的租户，这有助于系统扩展并保护其免受潜在的Dos攻击。这里Grafana很好地解释了Promtail(推荐的分发代理)如何处理数据。
• ingester：是Loki架构的关键组件。从distributors接收到的数据由ingester写入云云原生存储服务。ingesters还与查询协作，返回内存中的数据以响应读请求。
• Queriers：负责解析LogQL查询请求，并从ingesters或持久化存储中获取数据。
• query fronted：可选组件-提供API接口，可用于加速读处理。该组件通过将读请求排队、将大请求拆分为多个小请求以及缓存数据来优化读处理。

二、实验环境

---

1、k8s环境

版本v1.26.5，容器为containerd
二进制安装Kubernetes(K8s)集群(基于containerd)—从零安装教程（带证书）

主机名	IP	系统版本	安装服务
master01	10.10.10.21	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master02	10.10.10.22	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master03	10.10.10.23	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
node01	10.10.10.24	rhel7.5	nginx、kubelet、proxy
node02	10.10.10.25	rhel7.5	nginx、kubelet、proxy

2、Prometheus+Grafana环境

Prometheus+Grafana监控系统

主机名	IP	系统版本	安装服务
jenkins	10.10.10.10	rhel7.5	docker、Prometheus、Grafana

三、Loki安装

---

https://github.com/grafana/loki/releases
https://grafana.com/docs/loki/latest/installation/docker/

1、下载镜像

[root@jenkins ~]# docker pull grafana/loki:2.8.2
[root@jenkins ~]# docker pull grafana/promtail:2.8.2

push到本地harbor：

[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/promtail:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.8.2

2、启动loki

[root@jenkins ~]# mkdir -p /etc/loki/conf
[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/cmd/loki/loki-local-config.yaml -O loki-config.yaml
[root@jenkins ~]# mv loki-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name loki \
--privileged=true \
--restart always -p 3100:3100 \
-e TZ="Asia/Shanghai" \
-v /etc/localtime:/etc/localtime:ro \
-v /usr/share/zoneinfo/:/usr/share/zoneinfo/:ro \
-v /etc/loki/conf:/mnt/config \
grafana/loki:2.8.2 \
-config.file=/mnt/config/loki-config.yaml

[root@jenkins ~]# cat /etc/loki/conf/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

3、启动promtail

[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
[root@jenkins ~]# mv promtail-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name promtail \
  --privileged=true \
  --restart always \
  -e TZ="Asia/Shanghai" \
  -v /etc/localtime:/etc/localtime:ro \
  -v /etc/loki/conf/:/mnt/config \
  -v /var/log:/var/log \
  grafana/promtail:2.8.2 \
  -config.file=/mnt/config/promtail-config.yaml

[root@jenkins ~]# cat /etc/loki/conf/promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.10.10.10:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

4、测试结果

http://10.10.10.10:3100/metrics
http://10.10.10.10:3100/ready

5、winserver安装promtail

（1）下载promtail

（2）配置promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: D:/loki/config/positions.yaml

clients:
  - url: http://10.10.10.10:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: D:/loki/logs/*.out

（3）启动

./promtail-windows-amd64.exe --config.file=config/promtail-config.yaml

四、Loki插件

---

1、安装

https://github.com/donstephan/loki/blob/master/cmd/docker-driver/README.md#configure-the-default-logging-driver
https://grafana.com/docs/loki/latest/send-data/docker-driver/configuration/
优点：可以不用安装promtail，不用单独配置，能直接监控所有容器
缺点：如果loki挂了，其他微服务就会hang住，因此loki最好使用集群

[root@jenkins ~]# docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

更新版本（忽略）：

[root@jenkins ~]# docker plugin disable loki --force
[root@jenkins ~]# docker plugin upgrade loki grafana/loki-docker-driver:latest --grant-all-permissions
[root@jenkins ~]# docker plugin enable loki
[root@jenkins ~]# systemctl restart docker

2、配置daemon.json

全局收集配置，这里就不讲解单独收集（配置docker-compose）

# max-size=500m，意味着一个容器日志大小上限是500M
# max-file=3，意味着一个容器有三个日志，分别是id+.json、id+1.json、id+2.json
# loki-external-labels：添加额外label，不然无法区别不同服务器
[root@jenkins ~]# vim /etc/docker/daemon.json      # 添加配置
{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "http://10.10.10.10:3100/loki/api/v1/push",
        "loki-batch-size": "400",
        "max-size":"500m",
        "max-file":"3",
        "loki-external-labels": "instance=10.10.10.10,from=loki"
    }
}

[root@jenkins ~]# systemctl restart docker
[root@jenkins ~]# docker plugin ls
ID             NAME          DESCRIPTION           ENABLED
accc29da4254   loki:latest   Loki Logging Driver   true

注意：容器需要重新创建才能收集到日志

五、grafana查看

---

grafana可以使用13639模板

1、添加Loki数据源

2、查看本机日志

loki插件日志：

六、采集K8s日志

---

官网地址：https://grafana.com/docs/loki/latest/clients/promtail/installation/

1、上传镜像

发现使用2.8.2的镜像启动不成功，这里使用2.5.0版本

[root@jenkins ~]# docker pull grafana/promtail:2.5.0
[root@jenkins ~]# docker tag grafana/promtail:2.5.0 harbor.wielun.com/library/grafana/promtail:2.5.0
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.5.0

2、创建Endpoints

[root@master01 ~]# kubectl create ns logging
[root@master01 ~]# cat loki.yaml
apiVersion: v1
kind: Endpoints
metadata:
  name: loki
  namespace: logging
subsets:
  - addresses:
    - ip: 10.10.10.10
    ports:
    - port: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki
  namespace: logging
spec:
  clusterIP: None
  ports:
  - name: loki
    port: 3100
    protocol: TCP
    targetPort: 3100
  type: ClusterIP

[root@master01 ~]# kubectl apply -f loki.yaml

3、创建promtail

[root@master01 ~]# cat promtail.yaml
--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
  namespace: logging
  labels:
    app: promtail
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: harbor.wielun.com/library/grafana/promtail:2.5.0
        imagePullPolicy: IfNotPresent
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env:
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: logging
  labels:
    app: promtail
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
      - url: http://loki.logging.svc.cluster.local:3100/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
  labels:
    app: promtail
  namespace: logging
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount
  labels:
    app: promtail
  namespace: logging

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
  labels:
    app: promtail
  namespace: logging
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: default
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

[root@master01 ~]# kubectl apply -f promtail.yaml

4、查看结果

[root@master01 ~]# kubectl get pod -n logging
NAME                       READY   STATUS    RESTARTS   AGE
promtail-daemonset-2r8sv   1/1     Running   0          56s
promtail-daemonset-f4p8q   1/1     Running   0          56s
promtail-daemonset-tddsf   1/1     Running   0          56s
promtail-daemonset-wdn47   1/1     Running   0          56s
promtail-daemonset-gry5e   1/1     Running   0          56s

5、dashboard使用

https://grafana.com/grafana/dashboards/?dataSource=loki