官网
Grafana Loki documentation | Grafana Loki documentation
优势
- 轻量级、易部署:轻量级的日志聚合工具,易维护
- 标签索引存储:支持高效的日志查询和检索,使大规模分布系统中管理和查询日志数据变得更加方便
- 集成grafana:grafana loki 和grafana紧密集成,可以在grafana中直接展示和分析loki中的日志数据
- 灵活的查询语言:LogQL,帮助用户快速编写复杂的日志查询语句,实现快速检索和分析大量的日志数据
原理
Loki
- 用于日志聚合和存储的系统,旨在帮助用户处理大量的日志数据
- 使用标签化日志存储方式,类似于 Prometheus 的时间序列数据库,使得查询和筛选日志变得非常高效
- 用户可以将日志数据发送到 Loki 中,进行快速的查询和分析,并利用 Grafana 进行数据可视化
Promtail
- Promtail 是 Loki 的代理程序,用于收集和发送日志数据到 Loki 中
- 负责从各个源(例如文件、系统日志等)采集日志数据,并通过 Loki 的 API 将这些数据发送到 Loki 存储中
- 支持标签和模板,可以对日志进行标准化处理,并添加额外的元数据,以便更好地组织和检索日志数据
Loki部署
https://github.com/grafana/loki/releases
# 解压
mkdir -p /opt/loki/
cd /opt/loki/
unzip loki-linux-amd64.zip
配置详解:Grafana Loki 配置参数 |Grafana Loki 文档
# 配置
vim loki.yaml
target: "all"
auth_enabled: true
server:
http_listen_port: 3100
grpc_listen_port: 9095
http_listen_address: 0.0.0.0
grpc_listen_address: 0.0.0.0
graceful_shutdown_timeout: 60s
http_server_read_timeout: 60s
http_server_write_timeout: 60s
grpc_server_max_recv_msg_size: 10485760
grpc_server_max_send_msg_size: 10485760
table_manager:
retention_deletes_enabled: true
retention_period: 720h
ingester:
lifecycler:
join_after: 10s
observe_period: 5s
ring:
replication_factor: 1
kvstore:
store: memberlist
final_sleep: 0s
chunk_idle_period: 1m
chunk_target_size: 2048576
max_chunk_age: 1m
chunk_retain_period: 30s
wal:
enabled: true
dir: /loki/wal
checkpoint_duration: 1m
flush_on_shutdown: true
compactor:
working_directory: /opt/loki/compactor
shared_store: filesystem
retention_table_timeout: 10s
schema_config:
configs:
- from: 2024-01-01
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /opt/loki/boltdb-shipper-active
cache_location: /opt/loki/boltdb-shipper-cache
cache_ttl: 24h
shared_store: filesystem
filesystem:
directory: /opt/loki/chunks
common:
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
replication_factor: 1
limits_config:
max_cache_freshness_per_query: '5m'
reject_old_samples: true
reject_old_samples_max_age: 24h
query_range:
align_queries_with_step: true
max_retries: 5
parallelise_shardable_queries: true
cache_results: true
frontend:
compress_responses: true
max_outstanding_per_tenant: 2048
query_scheduler:
max_outstanding_requests_per_tenant: 1024
querier:
query_ingesters_within: 2h
配置开机自启
vim /etc/systemd/system/loki.service
[Unit]
Description=loki server
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/opt/loki/loki -config.file=/opt/loki/loki.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=loki
[Install]
WantedBy=default.target
systemctl daemon-reload && systemctl restart loki &&systemctl status loki
systemctl enable loki
netstat -tnlp(成功)
Promtail部署
Releases · grafana/loki (github.com)
# 解压
mkdir -p /opt/promtail/
cd /opt/promtail/
unzip promtail-linux-amd64.zip
# 配置
server:
http_listen_port: 9080
grpc_listen_port: 9096
http_server_read_timeout: 60s
http_server_write_timeout: 60s
positions:
filename: /opt/promtail/positions.yaml
sync_period: 10s
clients:
- url: http://IP:3100/loki/api/v1/push
tenant_id: test
batchsize: 2097152
external_labels:
env: "test"
scrape_configs:
- job_name: test
static_configs:
- targets:
- localhost
labels:
__path__: /xxx/xxx/*.log
job: test
host: IP
配置开机自启
vim /etc/systemd/system/promtail.service
[Unit]
Description=promtail server
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/opt/promtail/promtail -config.file=/opt/promtail/promtail.yaml
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail
[Install]
WantedBy=default.target
systemctl daemon-reload && systemctl restart promtail &&systemctl status promtail
systemctl enable promtail
netstat -tnlp(成功)
Granfan部署
看上篇
配置
Administration —— Data sources —— Add new data source —— Loki
查看
Home —— Explore
正则匹配的链接:https://grafana.com/docs/loki/latest/query/log_queries/