Grafana Loki 轻量级并轻松地对日志数据进行筛选、查询和分析

官网

Grafana Loki documentation | Grafana Loki documentation

优势

1. 轻量级、易部署：轻量级的日志聚合工具，易维护
2. 标签索引存储：支持高效的日志查询和检索，使大规模分布系统中管理和查询日志数据变得更加方便
3. 集成grafana：grafana loki 和grafana紧密集成，可以在grafana中直接展示和分析loki中的日志数据
4. 灵活的查询语言：LogQL，帮助用户快速编写复杂的日志查询语句，实现快速检索和分析大量的日志数据

原理

Loki

1. 用于日志聚合和存储的系统，旨在帮助用户处理大量的日志数据
2. 使用标签化日志存储方式，类似于 Prometheus 的时间序列数据库，使得查询和筛选日志变得非常高效
3. 用户可以将日志数据发送到 Loki 中，进行快速的查询和分析，并利用 Grafana 进行数据可视化

Promtail

1. Promtail 是 Loki 的代理程序，用于收集和发送日志数据到 Loki 中
2. 负责从各个源（例如文件、系统日志等）采集日志数据，并通过 Loki 的 API 将这些数据发送到 Loki 存储中
3. 支持标签和模板，可以对日志进行标准化处理，并添加额外的元数据，以便更好地组织和检索日志数据

Loki部署

https://github.com/grafana/loki/releases

# 解压
mkdir -p /opt/loki/
cd /opt/loki/
unzip loki-linux-amd64.zip

配置详解：Grafana Loki 配置参数 |Grafana Loki 文档 

# 配置
vim loki.yaml

target: "all"
auth_enabled: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  graceful_shutdown_timeout: 60s
  http_server_read_timeout: 60s
  http_server_write_timeout: 60s
  grpc_server_max_recv_msg_size: 10485760
  grpc_server_max_send_msg_size: 10485760

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

ingester:
  lifecycler:
    join_after: 10s
    observe_period: 5s
    ring:
      replication_factor: 1
      kvstore:
        store: memberlist
    final_sleep: 0s
  chunk_idle_period: 1m
  chunk_target_size: 2048576
  max_chunk_age: 1m
  chunk_retain_period: 30s

  wal:
    enabled: true
    dir: /loki/wal
    checkpoint_duration: 1m
    flush_on_shutdown: true

compactor:
  working_directory: /opt/loki/compactor
  shared_store: filesystem
  retention_table_timeout: 10s

schema_config:
  configs:
  - from: 2024-01-01
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /opt/loki/boltdb-shipper-active
    cache_location: /opt/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /opt/loki/chunks

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1

limits_config:
  max_cache_freshness_per_query: '5m'
  reject_old_samples: true
  reject_old_samples_max_age: 24h


query_range:
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

frontend:
  compress_responses: true
  max_outstanding_per_tenant: 2048

query_scheduler:
  max_outstanding_requests_per_tenant: 1024

querier:
  query_ingesters_within: 2h

 配置开机自启

vim /etc/systemd/system/loki.service

[Unit]
Description=loki server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/opt/loki/loki -config.file=/opt/loki/loki.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=loki
[Install]
WantedBy=default.target


systemctl daemon-reload  &&  systemctl restart loki  &&systemctl status loki
systemctl enable loki

netstat -tnlp（成功） 

 Promtail部署

Releases · grafana/loki (github.com)

# 解压
mkdir -p /opt/promtail/
cd /opt/promtail/
unzip promtail-linux-amd64.zip

# 配置

server:
    http_listen_port: 9080
    grpc_listen_port: 9096
    http_server_read_timeout: 60s
    http_server_write_timeout: 60s
positions:
    filename: /opt/promtail/positions.yaml
    sync_period: 10s

clients:
  - url: http://IP:3100/loki/api/v1/push
    tenant_id: test
    batchsize: 2097152
    external_labels:
      env: "test"
scrape_configs:
  - job_name: test
    static_configs:
      - targets:
        - localhost
        labels:
          __path__: /xxx/xxx/*.log
          job: test
          host: IP

 配置开机自启

vim /etc/systemd/system/promtail.service

[Unit]
Description=promtail server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/opt/promtail/promtail -config.file=/opt/promtail/promtail.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail
[Install]
WantedBy=default.target

systemctl daemon-reload  &&  systemctl restart promtail  &&systemctl status promtail
systemctl enable promtail 

netstat -tnlp（成功）

Granfan部署

看上篇

配置

Administration —— Data sources —— Add new data source —— Loki

 查看

Home —— Explore

正则匹配的链接：https://grafana.com/docs/loki/latest/query/log_queries/