CentOS7搭建日志系统Loki

CentOS7搭建日志系统Loki

架构1：Loki+promtail+grafana
架构2：Loki+logstash+grafana
各组件功能：
loki：Loki is the logging engine.
Promtail：Promtail sends logs to Loki.
Grafana：The Analitics platform for all your metrics
Logstash：集中、转换和存储数据

架构1：Loki+Promtail+Grafana

本地安装Loki和Promtail

1、下载
https://github.com/grafana/loki/releases/download/v2.1.0/loki-linux-amd64.zip
https://github.com/grafana/loki/releases/download/v2.1.0/promtail-linux-amd64.zip
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml
2、安装
mkdir /opt/loki
unzip loki-linux-amd64.zip -d /opt/loki
unzip promtail-linux-amd64.zip -d /opt/loki
3、配置Loki
vim /opt/loki/loki-local-config.yaml


4、配置Pormtail
收集messages文件信息，lables设置3个，分别是varlogs、host、path
vim /opt/loki/promtail-local-config.yaml

5、启动Loki和Pormtail
cd /opt/loki
nohup ./loki-linux-amd64 -config.file=loki-local-config.yaml > log-loki.file 2>&1 &
nohup ./promtail-linux-amd64 -config.file=promtail-local-config.yaml > log-promtail.file 2>&1 &

安装部署Grafana

下载
wget https://dl.grafana.com/oss/release/grafana-7.3.7-1.x86_64.rpm
yum install grafana-7.3.7-1.x86_64.rpm
启动服务
systemctl start grafana-server.service
7、Grafana添加Loki数据源
访问grafana： http://localhost:3000/
在Configuration添加Data Source，


URL填Loki服务器IP以及端口

Grafana中查看Loki日志

点击Explore，选择Loki，选择Log Lables，数据出现

|= "clock"筛选含有clock字段的日志行

查询基本语法：
= 等于
!=
=~
!~

架构2：Loki+Logstash+Grafana

安装Loki+Grafana

同上

安装Logstash

1、下载
https://artifacts.elastic.co/downloads/logstash/logstash-7.10.2-x86_64.rpm
2、安装
rpm -ivh logstash-7.10.2-x86_64.rpm
3、部署
[root@loki ~]# vim /etc/logstash/logstash.yml
node.name: loki
path.data: /var/lib/logstash
pipeline.workers: 8
pipeline.batch.size: 1000
pipeline.batch.delay: 50
pipeline.ordered: auto
log.level: info
path.logs: /var/log/logstash
[root@loki ~]# vim /etc/logstash/jvm.options
-Xms4g
-Xmx4g
[root@loki ~]# cp /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/
[root@loki ~]# vim /etc/logstash/conf.d/logstash-sample.conf
input {
file {
path => “/var/log/messages”
start_position => “beginning”
sincedb_path => “/dev/null”
type => “syslog”
}
filter {
mutate {
add_field => {
“job” => “logstash”
}
}
}
output {
loki {
url => “http://localhost:3100/loki/api/v1/push”
}
}

4、测试标准输入和标准输出
[root@loki ~]# cd /usr/share/logstash/bin/
./logstash -e ‘input { stdin {}} output { stdout {codec => rubydebug}}’
输入123456
输出：
{
“message” => “123456”,
“@version” => “1”,
“host” => “loki”,
“@timestamp” => 2020-08-11T02:39:48.704Z
}
5、验证配置
[root@loki ~]# cd /usr/share/logstash/bin/
[root@loki bin]# ./logstash -f /etc/logstash/conf.d/logstash-sample.conf --config.test_and_exit
6、开启服务
[root@loki bin]# systemctl start logstash

Grafana中查看Loki日志