How to install Grafana-Loki to query logs from your k8s servers?
Requirements:
System: ubuntu
Helm3 (Assuming installed)
Grafana
Loki
Promtail (send data[logs] to Loki)
Step to install Grafana-loki
- Install Grafana on k8s:
Refer: https://grafana.com/docs/grafana/latest/setup-grafana/installation/kubernetes/
- Install data source – Loki by helm3
Refer: https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/
Notice:
After installed, You can get a gateway address of Loki, which is used by Promtail later
***********************************************************************
Sending logs to Loki
***********************************************************************
Loki has been configured with a gateway (nginx) to support reads and writes from a single component.
You can send logs from inside the cluster using the cluster DNS:
http://loki-gateway.default.svc.cluster.local/loki/api/v1/push
- Install Promtail by helm3
Refer: https://grafana.com/docs/loki/latest/send-data/promtail/installation/
Notice:
Put your Loki-Gateway address here, For Loki-Auth required, Add X-Scope-OrgId: foo [Important]
```yaml
# values.yaml
config:
# publish data to loki
clients:
- url: http://loki-gateway/loki/api/v1/push
X-Scope-OrgId: foo
```
Finally, All Requirements are done.
Now you can use Grafana to see all logs you want, But set up Data sources first.
The Problems may appear && How to deal:
- Label amounts out of limit
# from Promtail you can get the error like below
error: loki has 16 label names; limit 15
Solution: Update your values of Promtail, Try official values for helm install Promtail
- HTTP 401
level=error ts=2024-09-05T07:53:52.821695784Z caller=client.go:430 component=client host=loki-gateway.default.svc.cluster.local msg="final error sending batch" status=401 tenant= error="server returned HTTP status 401 Unauthorized (401): no org id"
Solution: Add X-Scope-OrgId to values.yaml of Promtail
If this happening when you add data source by using Grafana, Also, add X-Scope-OrgId to Custom Header