loki+promtail+grafana日志收集

@王先生1

已于 2025-03-21 14:30:30 修改

阅读量1k

点赞数 12

文章标签： grafana loki promtail nginx

于 2024-04-26 13:54:30 首次发布

本文链接：https://blog.csdn.net/qq_44637753/article/details/138216334

版权

loki+promtail+grafana日志收集

一、在 172.200.171.140 部署 Grafana 和 Loki
二、在 172.200.171.143 部署 Promtail
三、关键配置说明
- 1. 日志路径权限
- 2. 正则表达式优化
四、验证服务状态
- 1. 检查 Loki 和 Promtail 服务
- 2、查看日志是否推送成功
五、Grafana 配置
- 1、访问 http://172.200.171.140:3000 登录（默认账号 admin/admin）。
常见问题排查
nginx 日志配置文件
K8S 容器日志收集

一、在 172.200.171.140 部署 Grafana 和 Loki

# 下载 Loki 二进制包（替换版本号）
LOKI_VERSION="2.8.0"
wget https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
ln -s /u01/loki-pro/loki-linux-amd64 /usr/bin/loki
chmod a+x /usr/bin/loki

# 创建配置文件 /etc/loki/loki-config.yaml
sudo mkdir -p /etc/loki
sudo tee /etc/loki/loki-config.yaml <<EOF
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100
schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h
ruler:
  alertmanager_url: http://localhost:9093
EOF

# 创建 Systemd 服务文件 /etc/systemd/system/loki.service
sudo tee /etc/systemd/system/loki.service <<EOF
[Unit]
Description=Loki Log Aggregation System
After=network.target

[Service]
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-config.yaml
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# 启动服务
sudo systemctl daemon-reload
sudo systemctl start loki
sudo systemctl enable loki

二、在 172.200.171.143 部署 Promtail

1. 安装 Promtail

# 下载 Promtail 二进制包（替换版本号）
PROMTAIL_VERSION="2.8.0"
wget https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
ln -s /u01/loki-pro/promtail /usr/bin/promtail
chmod a+x /usr/bin/promtail

2. 创建配置文件 /etc/promtail/promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://172.200.171.140:3100/loki/api/v1/push

scrape_configs:
  - job_name: jxb_logs
    static_configs:
      - targets: [localhost]
        labels:
          job: jxb_app
          __path__: /u01/log/comm-point/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?P<level>\w+)\s+\[(?P<thread>\w+)\s*\]\s+\[(?P<class>[\w\.\s]+)\]\s+(?P<line>\d+)\s+:\s+(?P<message>.*)$'
      - timestamp:
          format: "2006-01-02 15:04:05,000"
          source: timestamp
      - labels:
          level: "{{.level}}"
          class: "{{.class}}"
          thread: "{{.thread}}"
      - output:
          source: message
  - job_name: jxbpro_logs
    static_configs:
      - targets: [localhost]
        labels:
          job: jxbpro_app
          __path__: /u01/log/comm-pro/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?P<level>\w+)\s+\[(?P<thread>\w+)\s*\]\s+\[(?P<class>[\w\.\s]+)\]\s+(?P<line>\d+)\s+:\s+(?P<message>.*)$'
      - timestamp:
          format: "2006-01-02 15:04:05,000"
          source: timestamp
      - labels:
          level: "{{.level}}"
          class: "{{.class}}"
          thread: "{{.thread}}"
      - output:
          source: message

3. 创建 Systemd 服务文件 /etc/systemd/system/promtail.service

sudo tee /etc/systemd/system/promtail.service <<EOF
[Unit]
Description=Promtail Log Shipper
After=network.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-config.yaml
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# 启动服务
sudo systemctl daemon-reload
sudo systemctl start promtail
sudo systemctl enable promtail

##调试模式
./promtail -config.file=promtail-config.yaml -print-config-stderr

三、关键配置说明

1. 日志路径权限

确保 Promtail 有权限读取日志文件：

chmod -R 755 /var/log/jxb

2. 正则表达式优化

针对日志格式中的特殊字符（例如方括号和空格）调整正则表达式：

expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(?P<level>\w+)\s+\[(?P<thread>\w+)\s*\]\s+\[(?P<class>[\w\.\s]+)\]\s+(?P<line>\d+)\s+:\s+(?P<message>.*)$'

四、验证服务状态

1. 检查 Loki 和 Promtail 服务

# 检查 Loki
curl http://172.200.171.140:3100/ready

# 检查 Promtail
curl http://172.200.171.143:9080/ready

2、查看日志是否推送成功

#在 Loki 中查询日志：

curl -G http://172.200.171.140:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="jxb_app"}' \
  --data-urlencode 'limit=5'

五、Grafana 配置

1、访问 http://172.200.171.140:3000 登录（默认账号 admin/admin）。

添加 Loki 数据源：

Name: Loki

URL: http://localhost:3100

使用 LogQL 查询日志：

logql
复制
{job="jxb_app"} | line_format "{{.class}} [{{.level}}] {{.message}}"

常见问题排查

1. Promtail 无法读取日志

# 检查文件权限
ls -l /var/log/jxb


# 查看 Promtail 日志
journalctl -u promtail -f

2. Loki 存储配置

修改 /etc/loki/loki-config.yaml 指定持久化存储路径：

common:
  path_prefix: /data/loki  # 替换为实际存储路径
  storage:
    filesystem:
      chunks_directory: /data/loki/chunks
      rules_directory: /data/loki/rules

3. 多行日志处理

如果日志包含堆栈信息，在 Promtail 配置中添加 multiline 阶段：

pipeline_stages:
- multiline:
    firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}'
    max_wait_time: 3s

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.10.222:3100/loki/api/v1/push

scrape_configs:
- job_name: nginx
  static_configs:
  - targets:
      - 192.168.10.222
    labels:
      job: nginx_logs
      __path__: /etc/nginx/logs/access.log
  - targets:
      - 192.168.10.222
    labels:
      job: sys_logs
      __path__: /var/log/syslog
  - targets:
      - 192.168.10.222
    labels:
      job: auth_logs
      __path__: /var/log/auth.log

nginx 日志配置文件

log_format json '{"@timestamp":"$time_iso8601",'
              '"clientip":"$remote_addr",'
              '"clientip2":"$proxy_protocol_addr",'
              '"responsetime":$request_time,'
              '"upstreamtime":"$upstream_response_time",'
              '"upstreamhost":"$upstream_addr",'
              '"domain":"$host",'
              '"url":"$uri",'
              '"referer":"$http_referer",'
              '"status":"$status"}';

access_log  /var/log/nginx/access.log  json;

![在这里插入图片描述](https://img-blog.csdnimg.cn/direct/b67f176979ed

K8S 容器日志收集

server:
  log_level: info
  http_listen_port: 3101

client:
  url: http://loki-0.loki-headless.kube-ops.svc.cluster.local:3100/loki/api/v1/push
  

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - docker: {} # 要先用docker 解析 容器日志
      - match:
          selector: '{namespace="default",pod=~"sk-.+"}'
          stages:
          - json:
              expressions:
                caller: caller
                level: level
                msg: msg
                span_id: span_id
                time: time
          - labels:
              level: null
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - namespace
        - app
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
        - __meta_kubernetes_pod_container_name
        target_label: __path__