Grafana Loki官方文档翻译 Get Start部分

关于Loki能搜到的文章不多，大多数都是专注于如何配置Loki的。自己动手丰衣足食，对Loki官方文档中对Loki架构进行介绍的部分进行了翻译。
官方文档链接：https://grafana.com/docs/loki/latest/
本博文的主要内容为文档的“GET START部分”，机翻+人工校对，多有不足，请见谅

Loki官方文档翻译

Grafana Loki documentation

Grafana Loki is a set of components that can be composed into a fully featured logging stack.

Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.
A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

Grafana Loki是一套组件，可以组成一个功能齐全的日志堆栈。

与其它日志系统不同，Loki的思想是：仅索引日志的元数据——标签（就像prometheus标签）。日志数据本身会被压缩并存储在对象存储，例如Amazon Simple Storage Service (S3)或者Google Cloud Storage (GCS) 的chunks里，甚至直接存储在文件系统。

小索引和高度压缩的chunk简化了操作，并且显著降低了Loki的成本。

Get start

Loki overview Loki概述

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of pull.

Loki is designed to be very cost effective and highly scalable. Unlike other logging systems, Loki does not index the contents of the logs, but only indexes metadata about your logs as a set of labels for each log stream.

A log stream is a set of logs which share the same labels. Labels help Loki to find a log stream within your data store, so having a quality set of labels is key to efficient query execution.

Log data is then compressed and stored in chunks in an object store such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even, for development or proof of concept, on the filesystem. A small index and highly compressed chunks simplify the operation and significantly lower the cost of Loki.

Loki是一个水平可扩展、高可用、多租户的日志聚合系统，灵感来自Prometheus。与prometheus不同的是，Loki专注于日志而不是度量（metrics），并且通过推送（push）而不是拉取（pull）来收集日志。

Loki的设计使得它具有很高的成本效益和可扩展性。与其他日志系统不同，Loku不索引日志的内容，而只索引关于日志的元数据，即每个日志流的标签集。

一个日志流是一组共享相同标签（Label）的日志。标签帮助Loki在数据存储中查找日志流，因此，拥有高质量的标签集是高效查询执行的关键。

然后，将日志数据压缩并存储在对象存储中，例如Amazon Simple Storage Service（S3）或Google Cloud Storage（GCS），或者甚至对于开发或概念验证，也可以存储在文件系统中。小型索引和高度压缩的chunks简化了操作，显著降低了Loki的成本。

A typical Loki-based logging stack consists of 3 components:

• Agent - An agent or client, for example Promtail, which is distributed with Loki, or the Grafana Agent. The agent scrapes logs, turns the logs into streams by adding labels, and pushes the streams to Loki through an HTTP API.
• Loki - The main server, responsible for ingesting and storing logs and processing queries. It can be deployed in three different configurations, for more information see deployment modes.
• Grafana for querying and displaying log data. You can also query logs from the command line, using LogCLI or using the Loki API directly.

基于Loki的典型日志堆栈由三个组件组成：

• Agent：一个代理（Agent）或客户端（client），例如与Loki一起发布的Promtail或Grafana Agent。代理负责抓取日志，同各国添加标签将日志转换为日志流，并通过HTTP API将日志流推送到Loki。
• Loki：主服务器，负责摄入和存储日志并处理查询。它可以部署为三种不同配置。更多信息参考…
• Grafana：用于查询和显示日志数据。还可以使用LogCLI或者直接使用Loki API从命令行查询日志。

Loki features Loki特点

• Scalability - Loki is designed for scalability, and can scale from as small as running on a Raspberry Pi to ingesting petabytes a day. In its most common deployment, “simple scalable mode”, Loki decouples requests into separate read and write paths, so that you can independently scale them, which leads to flexible large-scale installations that can quickly adapt to meet your workload at any given time. If needed, each of Loki’s components can also be run as microservices designed to run natively within Kubernetes.
• Multi-tenancy - Loki allows multiple tenants to share a single Loki instance. With multi-tenancy, the data and requests of each tenant is completely isolated from the others. Multi-tenancy is configured by assigning a tenant ID in the agent.
• Third-party integrations - Several third-party agents (clients) have support for Loki, via plugins. This lets you keep your existing observability setup while also shipping logs to Loki.
• Efficient storage - Loki stores log data in highly compressed chunks. Similarly, the Loki index, because it indexes only the set of labels, is significantly smaller than other log aggregation tools. By leveraging object storage as the only data storage mechanism, Loki inherits the reliability and stability of the underlying object store. It also capitalizes on both the cost efficiency and operational simplicity of object storage over other storage mechanisms like locally attached solid state drives (SSD) and hard disk drives (HDD).
  The compressed chunks, smaller index, and use of low-cost object storage, make Loki less expensive to operate.
• LogQL, Loki’s query language - LogQL is the query language for Loki. Users who are already familiar with the Prometheus query language, PromQL, will find LogQL familiar and flexible for generating queries against the logs. The language also facilitates the generation of metrics from log data, a powerful feature that goes well beyond log aggregation.
• Alerting - Loki includes a component called the ruler, which can continually evaluate queries against your logs, and perform an action based on the result. This allows you to monitor your logs for anomalies or events. Loki integrates with Prometheus Alertmanager, or the alert manager within Grafana.
• Grafana integration - Loki integrates with Grafana, Mimir, and Tempo, providing a complete observability stack, and seamless correlation between logs, metrics and traces.

• 可扩展性：Loki为可扩展性而设计，其规模可以从在Raspberry Pi上运行到每天处理数百TB的数据。在其最常见的部署中，“简单可扩展模式”下，Loki将请求分解为独立的读取和写入路径，以便您可以独立地扩展它们，从而实现了灵活的大规模安装，可以随时快速适应您的工作负载。如有需要，Loki的每个组件还可以作为旨在在Kubernetes内部运行作为微服务的组件来运行。
• 多租户：Loki允许多个租户共享单个Loki实例。通过多租户，每个租户的数据和请求与其他租户完全隔离。通过在代理中分配租户ID来配置多租户。
• 第三方集成：一些第三方代理（客户端）通过插件支持Loki。这使您能够保持现有的可观察性设置，同时还能将日志发送到Loki。
• 高效存储：Loki将日志数据存储在高度压缩的块中。类似地，由于Loki的索引仅对标签进行索引，因此它比其他日志聚合工具小得多。通过利用对象存储作为唯一的数据存储机制，Loki继承了底层对象存储的可靠性和稳定性。它还利用了对象存储相对于其他存储机制（如本地附加的固态驱动器（SSD）和硬盘驱动器（HDD））在成本效益和操作简便性方面的优势。压缩块、较小的索引和对低成本对象存储的使用使Loki的运营成本更低。
• LogQL：LogQL是Loki的查询语言。对于已经熟悉Prometheus查询语言PromQL的用户来说，LogQL将为他们提供熟悉且灵活的查询日志的方式。该语言还便于从日志数据中生成度量标准，这是强大而卓越的功能，远远超越了日志聚合的功能。
• 告警：Loki 包含一个名为“ruler”的组件，该组件可以根据您的日志持续评估查询，并根据结果执行操作。这允许您监视您的日志以检测异常或事件。Loki 与 Prometheus Alertmanager 或 Grafana 中的告警管理器集成。
• Grafana集成：Loki 与 Grafana、Mimir 和 Tempo 集成，提供完整的可观察性堆栈，以及日志、指标和跟踪之间的无缝关联。

Loki architecture Loki架构

Multi-tenacy 多租户

All data, both in memory and in long-term storage, may be partitioned by a tenant ID, pulled from the X-Scope-OrgID HTTP header in the request when Grafana Loki is running in multi-tenant mode. When Loki is not in multi-tenant mode, the header is ignored and the tenant ID is set to “fake”, which will appear in the index and in stored chunks.

在多租户模式下运行时，Grafana Loki可以从请求的X-Scope-OrgID HTTP header中提取租户ID，并使用该租户ID对内存中和长期存储中的所有数据进行分区。

在非多租户模式下，该header 将被忽略，租户ID将设置为"fake"，该ID将出现在索引和存储块中。

Chunk Format Chunk格式

  ----------------------------------------------------------------------------
  |                        |                       |                         |
  |     MagicNumber(4b)    |     version(1b)       |      encoding (1b)      |
  |                        |                       |                         |
  ----------------------------------------------------------------------------
  |                      #structuredMetadata (uvarint)                       |
  ----------------------------------------------------------------------------
  |      len(label-1) (uvarint)      |          label-1 (bytes)              |
  ----------------------------------------------------------------------------
  |      len(label-2) (uvarint)      |          label-2 (bytes)              |
  ----------------------------------------------------------------------------
  |      len(label-n) (uvarint)      |          label-n (bytes)              |
  ----------------------------------------------------------------------------
  |                      checksum(from #structuredMetadata)                  |
  ----------------------------------------------------------------------------
  |           block-1 bytes          |           checksum (4b)               |
  ----------------------------------------------------------------------------
  |           block-2 bytes          |           checksum (4b)               |
  ----------------------------------------------------------------------------
  |           block-n bytes          |           checksum (4b)               |
  ----------------------------------------------------------------------------
  |                           #blocks (uvarint)                              |
  ----------------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
  ----------------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
  ----------------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
  ----------------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
  ----------------------------------------------------------------------------
  |                          checksum(from #blocks)                          |
  ----------------------------------------------------------------------------
  | #structuredMetadata len (uvarint) | #structuredMetadata offset (uvarint) |
  ----------------------------------------------------------------------------
  |     #blocks len (uvarint)         |       #blocks offset (uvarint)      |
  ----------------------------------------------------------------------------

mint and maxt describe the minimum and maximum Unix nanosecond timestamp, respectively.

The structuredMetadata section stores non-repeated strings. It is used to store label names and label values from structured metadata. Note that the labels strings and lengths within the structuredMetadata section are stored compressed.

mint和maxt分别表述了Unix纳秒时间戳的最小值和最大值。

structuredMetadata存储非重复的字符串，用于存储来自结构化元数据的标签名称和标签值。注意，structuredMetadata部分中的标签字符串和长度是压缩存储的。

Block Format Block格式

A block is comprised of a series of entries, each of which is an individual log line.

Note that the bytes of a block are stored compressed. The following is their form when uncompressed:

一个Block由一系列条目组成，每个条目都是一个单独的日志行

注意：块的字节是压缩存储的。以下是在未压缩时的形式：

-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-1 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-2 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-3 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-n bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------

ts is the Unix nanosecond timestamp of the logs, while len is the length in bytes of the log entry.

Symbols store references to the actual strings containing label names and values in the structuredMetadata section of the chunk.

ts是日志的Unix纳秒时间戳，len是日志条目的字节长度。

symbols存储引用，这个引用是对于chunk的structuredMetadata中包含的标签名称和值的实际字符串的引用

Storage 存储

Single Storge 单独存储

Loki stores all data in a single object storage backend. This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple, not to mention where all current and future development lies. This mode uses an adapter called boltdb_shipper to store the index in object storage (the same way we store chunks).

Loki将所有数据存储在单个对象存储后端中。这种操作模式在Loki 2.0中普遍可用，并且速度快、成本低、简单，更不用说当前和未来的所有开发都在此基础上运行。这种模式使用名为boltdb_shipper的适配器将索引（index）存储在对象存储中（与我们存储chunks的方式相同）。

Deprecated: Multi-store 不推荐：多存储

The chunk store is Loki’s long-term data store, designed to support interactive querying and sustained writing without the need for background maintenance tasks. It consists of:

chunk store是Loki的长期数据存储，旨在支持交互式查询和持续写入，而无需后台维护任务。它由以下部分组成：

• An index for the chunks. This index can be backed by:
  • Amazon DynamoDB
  • Google Bigtable
  • Apache Cassandra

chunks的索引。这个索引可以通过以下方式实现

• A key-value (KV) store for the chunk data itself, which can be:
  • Amazon DynamoDB
  • Google Bigtable
  • Apache Cassandra
  • Amazon S3
  • Google Cloud Storage

一个键值存储器，用于存储chunk data本身。该存储器可以是

Unlike the other core components of Loki, the chunk store is not a separate service, job, or process, but rather a library embedded in the two services that need to access Loki data: the ingester and querier.

与其他Loki的核心组件不同，chunk store不是单独的服务、作业或进程，而是嵌入在需要访问Loki数据的两个服务中：ingester（数据摄取器） 和querier（查询器）

The chunk store relies on a unified interface to the “NoSQL” stores (DynamoDB, Bigtable, and Cassandra) that can be used to back the chunk store index. This interface assumes that the index is a collection of entries keyed by:

• A hash key. This is required for all reads and writes.
• A range key. This is required for writes and can be omitted for reads, which can be queried by prefix or range.

chunk store依赖于一个统一的接口来和与“NoSQL”存储（DynamoDB、Bigtable和Cassandra）进行交互，这个接口可用于支持chunk store index。该接口假定索引是一组以以下键为key的条目：

• 哈希key：这是所有读取和写入操作必须的
• range key：这是写入操作必须的。读操作可以通过前缀或范围（range）进行查询，因此可以省略

The interface works somewhat differently across the supported databases:

• DynamoDB supports range and hash keys natively. Index entries are thus modelled directly as DynamoDB entries, with the hash key as the distribution key and the range as the DynamoDB range key.
• For Bigtable and Cassandra, index entries are modelled as individual column values. The hash key becomes the row key and the range key becomes the column key.

对于所支持的数据库，接口的工作方式有所不同：

• DynamoDB原生支持范围键和哈希键。因此，索引条目直接作为DynamoDB条目进行建模，其中哈希键作为分布键，范围作为DynamoDB范围键。
• 对于Bigtable和Cassandra，索引条目被建模为单个列值。哈希键成为行键，范围键成为列键。

A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Loki has evolved, mainly in an attempt to better load balance writes and improve query performance.

一系列schemas用于将读写操作中使用的匹配器（matchers）和标签集（label sets）映射到chunk store上的适当操作。随着Loki的发展，Schemas已经被添加，主要是为了更好地平衡写入和提升查询性能。

Read Path 读取路径

To summarize, the read path works as follows:

1. The querier receives an HTTP/1 request for data.
2. The querier passes the query to all ingesters for in-memory data.
3. The ingesters receive the read request and return data matching the query, if any.
4. The querier lazily loads data from the backing store and runs the query against it if no ingesters returned data.
5. The querier iterates over all received data and deduplicates, returning a final set of data over the HTTP/1 connection.’

总结读取路径的工作方式：

1. 查询器（querier）接收到一个HTTP/1的查询数据请求
2. querier将查询传递给所有内存数据的摄取器（ ingesters）
3. ingesters接收到读取请求并返回匹配查询的数据（如果有）
4. 如果没有ingesters返回数据，querier从后端存储中lazily加载数据，并在其上运行查询
5. 查询器迭代所有接收到的数据并进行去重，通过HTTP/1连结返回最终的数据集

Write Path 写入路径

To summarize, the write path works as follows:

1. The distributor receives an HTTP/1 request to store data for streams.
2. Each stream is hashed using the hash ring.
3. The distributor sends each stream to the appropriate ingesters and their replicas (based on the configured replication factor).
4. Each ingester will create a chunk or append to an existing chunk for the stream’s data. A chunk is unique per tenant and per labelset.
5. The distributor responds with a success code over the HTTP/1 connection.

总结写入路径的工作方式：

1. distributor接收到一个用于存储流数据的HTTP/1请求
2. 每个流都使用哈希环进行哈希
3. distributor将每个流发送到适当的ingesters及其副本（基于配置的复制因子）
4. 每个ingesters将为流数据创建一个chunk或附加到现有块中。每个租户和每个标签集都有一个唯一的块。
5. distributor通过HTTP/1响应发送一个成功代码。

Loki deployment modes Loki部署模式

Loki is a distributed system consisting of many microservices. It also has a unique build model where all of those microservices exist within the same binary.

Loki是一个由许多微服务组成的分布式系统。它还具有独特的构建模型，其中所有这些微服务都存在于同一个二进制文件中。

You can configure the behavior of the single binary with the -target command-line flag to specify which microservices will run on startup. You can further configure each of the components in the loki.yaml file.

可以使用-target命令行标志来配置单个二进制文件的行为，以指定哪些微服务将在启动时运行。还可以配置loki.yaml文件中的每个组件。

Because Loki decouples the data it stores from the software which ingests and queries it, you can easily redeploy a cluster under a different mode as your needs change, with minimal or no configuration changes.

由于Loki将存储的数据与ingests和queries解耦，因此随着需求的变化，可以轻松地以不同的模式重新部署集群，而无需进行任何配置更改或仅进行最小的更改。

Simple Scalable 简单扩展部署

The simple scalable deployment mode, is the preferred way to deploy Loki for most installations. The simple scalable deployment is the default configuration installed by the Loki Helm Chart. This deployment mode is the easiest way to deploy Loki at scale. It strikes a balance between deploying in monolithic mode or deploying each component as a separate microservice.

简单扩展部署模式是部署Loki的首选方式，适用于大多数安装。

简单可扩展部署是Loki Helm Chart默认安装的配置。这种部署模式是部署大规模Loki的最简单方式。它在整体部署模式和将每个组件作为单独的微服务进行部署之间找到了平衡。

Note: This deployment mode is sometimes referred to by the acronym SSD for simple scalable deployment, not to be confused with solid state drives. Loki uses an object store.

注：这种部署模式有时候通过缩写SSD来表示（意味简单 Simple 可扩展 Scalable 部署 deployment），不要与固态驱动器混淆。Loki使用对象存储？

Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets. These targets can be scaled independently, letting you customize your Loki deployment to meet your business needs for log ingestion and log query so that your infrastructure costs better match how you use Loki.

Loki的简单可扩展部署模式将执行路径分为读取、写入和后端目标（backend targets）这些目标可以独立扩展，使用户能够根据日志ingestion和日志query的业务需求定制loki部署，从而使基础设施成本更好地与您使用Loki的方式相匹配。

The simple scalable deployment mode can scale up to a few TBs of logs per day, however if you go much beyond this, the microservices mode will be a better choice for most users.

简单的可扩展部署模式可以每天扩展到TB级别的日志，但如果超越此规模，对于大多数用户来说，微服务模式将是更好的选择。

The three execution paths in simple scalable mode are each activated by appending the following arguments to Loki on startup:

在简单可扩展模式下，三种执行路径通过在启动Loki时附加以下参数来激活：

• -target=write - The write target is stateful and is controlled by a Kubernetes StatefulSet. It contains the following components: – Distributor – Ingester
• -target=read - The read target is stateless and can be run as a Kubernetes Deployment that can be scaled automatically. It contains the following components: – Query front end – Queriers
• -target=backend - The backend target is stateful, and is controlled by a Kubernetes StatefulSet. Contains the following components: – Compactor – Index gateways – Query scheduler – Ruler

• -target = write：写target是有状态的，由Kubernetes StatefulSet控制。它包含以下组件：– Distributor – Ingester
• -target = read：读target是无状态的，可作为能自动扩展的Kubernetes Deployment运行。它包含以下组件：Query front end – Queriers
• -target = backend：后端target是有状态的，由Kubernetes StatefulSet控制，包含以下组件：– Compactor – Index gateways – Query scheduler – Ruler

The simple scalable deployment mode requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.

SSD模式需要在Loki签名部署一个反向代理，将客户端API请求定向到读节点或写节点，Loki Helm图表包含一个默认的反向代理配置，使用Nginx。

Monolithic mode 单体部署

The simplest mode of operation is the monolithic deployment mode. You enable monolithic mode by setting the -target=all command line parameter. This mode runs all of Loki’s microservice components inside a single process as a single binary or Docker image.

最简单的操作模式是单体部署模式。通过设置-target = all命令行参数，即可启用单体模式。

在这种模式下，Loki的所有微服务组件作为一个二进制文件或Dcoker镜像在单个进程中运行。

Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day.

单块模式对于快速入门Loki进行实验以及每天最多约20GB的小读写卷很有用。

You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the ring section of the loki.yaml file to share state between all instances, but the recommendation is to use simple scalable mode if you need to scale your deployment.

通过使用共享对象存储以及配置Loki.yaml文件的ring部分以在所有实例之间共享状态，您可以将单体模式部署水平扩展到更多实例。但是，如果您需要扩展部署，建议使用简单的可扩展模式。

You can configure high availability by running two Loki instances using memberlist_config configuration and a shared object store and setting the replication_factor to 3. You route traffic to all the Loki instances in a round robin fashion.

您可以通过运行两个使用menberlist_config配置和共享对象存储运行两个loki实例，并将复制因子设置为3来配置高可用性。您以轮询的方式将流量路由到所有Loki实例。

Query parallelization is limited by the number of instances and the setting max_query_parallelism which is defined in the loki.yaml file.

查询并行化受到实例数量和在loki.yaml文件中定义的max_query_paralleism设置限制。

Microservices mode 微服务部署

The microservices deployment mode runs components of Loki as distinct processes. Each process is invoked specifying its target: For release 2.9 the components are:

微服务部署模式将Loki的组件作为不同的进程运行。每个进程都通过指定其target来调用。对于2.9版本，组件如下：

• Cache Generation Loader
• Compactor
• Distributor
• Index-gateway
• Ingester
• Ingester-Querier
• Overrides Exporter
• Querier
• Query-frontend
• Query-scheduler
• Ruler
• Table Manager (deprecated)

TIP: You can see the complete list of targets for your version of Loki by running Loki with the flag -list-targets, for example:

提示：您可以通过运行带有-list-targets标志的Loki来查看您版本的目标列表，例如：

docker run docker.io/grafana/loki:2.9.2 -config.file=/etc/loki/local-config.yaml -list-targets

Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case.

将运行组件作为单独的微服务能够提供更精细的粒度，让您将每个组件作为单独的微服务进行扩展，以更好地匹配您的特定用例。

Microservices mode deployments can be more efficient Loki installations. However, they are also the most complex to set up and maintain.

微服务模式部署可以更高效地安装Loki，然而，它们也是设置和维护起来最复杂的。

Microservices mode is only recommended for very large Loki clusters or for operators who need more precise control over scaling and cluster operations.

微服务模式仅适用于非常大的Loki集群或需要对缩放和集群操作有更精确控制的运营商。

Microservices mode is designed for Kubernetes deployments. A community-supported Helm chart is available for deploying Loki in microservices mode.

微服务模式转为Kubernetes部署而设计。社区支持的Helm图表可用于在微服务模式下部署Loki。

Loki Components Loki组件

Distributor

The distributor service is responsible for handling incoming streams by clients. It’s the first stop in the write path for log data. Once the distributor receives a set of streams, each stream is validated for correctness and to ensure that it is within the configured tenant (or global) limits. Valid chunks are then split into batches and sent to multiple ingesters in parallel.

Distributor服务器负责处理来自客户端的入站流。对于日志数据来说，这是写入路径的第一站。一旦distributor收到一组流，就会对每个流进行验证，确保其正确性并确保它在配置的租户（或全局）限制内。然后，有效的chunks被分成批处理，并并行发送给多个ingesters。

It is important that a load balancer sits in front of the distributor in order to properly balance traffic to them.

负载均衡器必须位于distributor的前面，以正确地平衡向它们的流量。这一点很重要。

The distributor is a stateless component. This makes it easy to scale and offload as much work as possible from the ingesters, which are the most critical component on the write path. The ability to independently scale these validation operations mean that Loki can also protect itself against denial of service attacks (either malicious or not) that could otherwise overload the ingesters. They act like the bouncer at the front door, ensuring everyone is appropriately dressed and has an invitation. It also allows us to fan-out writes according to our replication factor.

distributor是一个无状态的组件，这使得它可以轻松得扩展并从ingesters卸载尽可能多得工作，ingesters是写入路径上最关键得组件，能够独立地扩展这些validation operations意味着Loki还可以防止可能过度使用ingesters得拒绝服务攻击。它们就像在前门守卫的门卫，确保每个人穿着得体并持有邀请函，它还允许我们根据replication factor（复制因子）进行写入扩展。

Validation

The first step the distributor takes is to ensure that all incoming data is according to specification. This includes things like checking that the labels are valid Prometheus labels as well as ensuring the timestamps aren’t too old or too new or the log lines aren’t too long.

distributor采取的第一步是确保所有传入的数据符合规范，这包括检查标签是否为有效的Prometheus标签，以及确保时间戳不是太旧或者太新，日志行不是太长。

Preprocessing

Currently the only way the distributor mutates incoming data is by normalizing labels. What this means is making {foo="bar", bazz="buzz"} equivalent to {bazz="buzz", foo="bar"}, or in other words, sorting the labels. This allows Loki to cache and hash them deterministically.

目前，distributor对传入数据进行变异的唯一方式是通过规范化标签。这意味着将{foo=“bar”, bazz=“buzz”}等效于{bazz=“buzz”, foo=“bar”}，换句话说，就是对标签进行排序。这使得Loki能够确定性地缓存和哈希它们。

Rate limiting

The distributor can also rate limit incoming logs based on the maximum per-tenant bitrate. It does this by checking a per tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/second before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to (10MB / 20 distributors) = 500KB/s! This is how global limits allow much simpler and safer operation of the Loki cluster.

Note: The distributor uses the ring component under the hood to register itself amongst its peers and get the total number of active distributors. This is a different “key” than the ingesters use in the ring and comes from the distributor’s own ring configuration.

distributor还可以根据每个租户的最大比特率限制传入日志。这是通过检查每个租户地限制并将其除以当前经distributor地数量来实现地。这允许在集群级别上为每个租户指定速率限制，并使我们能够扩展或缩小distributor，并相应地调整每个distributor地限制。

例如，假设有10个distributor，租户A有一个10MB的速率限制，每个distributor在限制之前最多允许1MB/秒。

现在另一个大型租户加入集群，我们需要启动另外10个distributor。现在20个distributor将调整他们对租户A的速率限制为（10MB / 20个distributor）= 500KB/秒！

这就是全局限制如何使Loki集群的操作更加简单和安全。

注意：distributor在后台使用ring组件在同类节点中进行注册，并获取活动distributor的总数。这与ingesters在ring中使用的key不同，它来自distributor自己的环配置。

Forwarding

Once the distributor has performed all of its validation duties, it forwards data to the ingester component which is ultimately responsible for acknowledging the write.

一旦 distributor完成了其所有的验证职责，它就会将数据转发给负载确认写入的ingester组件。

Replication factor

In order to mitigate the chance of losing data on any single ingester, the distributor will forward writes to a replication_factor of them. Generally, this is 3. Replication allows for ingester restarts and rollouts without failing writes and adds additional protection from data loss for some scenarios. Loosely, for each label set (called a stream) that is pushed to a distributor, it will hash the labels and use the resulting value to look up replication_factor ingesters in the ring (which is a subcomponent that exposes a distributed hash table). It will then try to write the same data to all of them. This will error if less than a quorum of writes succeed. A quorum is defined as floor(replication_factor / 2) + 1. So, for our replication_factor of 3, we require that two writes succeed. If less than two writes succeed, the distributor returns an error and the write can be retried.

为了减少任何单个 ingester上丢失数据的可能性，distributor会将写入操作转发给多个 ingester。一般来说，是3个。Replication允许Ingester重新启动和滚动升级，而不会导致写入失败，并为某些场景增加了额外的数据丢失保护。对于推送到一个distributor的每个标签集（称为一个流），它会对标签进行哈希处理，并使用结果值在ring中查找相应的replication_factor ingesters（Ring是一个子组件，它公开了一个分布式哈希表）然后，它会尝试将相同的数据写入所有找到的Ingesters。如果少于一个Quorum（法定人数）的写入操作成功，Distributor会返回错误，并且可以重试写入操作。Quorum被定义为 floor(replication_factor / 2) + 1。因此，当 replication_factor 为 3，需要两个写操作成功。如果少于两个写操作成功，分发器会返回错误，重新执行写操作。

Caveat: If a write is acknowledgd by 2 out of 3 ingesters, we can tolerate the loss of one ingester but not two, as this would result in data loss.

注意：如果三个ingesters中的两个ingesters确认了写操作，则可以容忍一个ingesters的loss。但是不能有两个ingesters loss，这回导致数据丢失。

Replication factor isn’t the only thing that prevents data loss, though, and arguably these days its main purpose is to allow writes to continue uninterrupted during rollouts & restarts. The ingester component now includes a write ahead log which persists incoming writes to disk to ensure they’re not lost as long as the disk isn’t corrupted. The complementary nature of replication factor and WAL ensures data isn’t lost unless there are significant failures in both mechanisms (i.e. multiple ingesters die and lose/corrupt their disks).

Replication factor不是防止数据丢失的唯一因素。它的主要作用是在rollouts & restarts（发布和重启）期间写入操作能够不间断地继续进行。

ingester组件还包含一个写前日志（write ahead log ，WAL），WAL 的功能是确保所有传入的写操作在写入磁盘之前先记录下来。只要磁盘没有损坏，写操作就不会丢失。

复制因子（replication factor）和 WAL 是互补的。复制因子确保数据冗余和可用性的方法， WAL 确保数据的持久性。除非这两个机制都发生重大故障（例如多个 ingester 死亡并且它们的磁盘丢失或损坏），否则数据不会丢失。

Hashing

Distributors use consistent hashing in conjunction with a configurable replication factor to determine which instances of the ingester service should receive a given stream.

Distributors使用一致性哈希+自定义replication factor决定哪些ingester服务接受输入流。

A stream is a set of logs associated to a tenant and a unique labelset. The stream is hashed using both the tenant ID and the labelset and then the hash is used to find the ingesters to send the stream to.

一个流是一组和租户和唯一标签集相关联的日志。使用租户ID和标签集对流进行哈希，然后使用这个哈希值来确定向哪个ingesters发送当前流。

A hash ring stored in Consul is used to achieve consistent hashing; all ingesters register themselves into the hash ring with a set of tokens they own. Each token is a random unsigned 32-bit number. Along with a set of tokens, ingesters register their state into the hash ring. The state JOINING, and ACTIVE may all receive write requests, while ACTIVE and LEAVING ingesters may receive read requests. When doing a hash lookup, distributors only use tokens for ingesters who are in the appropriate state for the request.

Consul 中存储的哈希环用于实现一致性哈希。所有ingesters都会用自己的令牌集合将自己注册到哈希环宏。每个令牌都是一个随机的无符号32位数字。在注册自己的令牌的同时，ingesters还会将自己的状态注册到哈希环中，状态JOINING,和ACTIVE可以接受写请求，状态ACTIVE 和 LEAVING 可以接受读请求。在进行哈希查找时，distributor只使用处于相应状态的 ingesters 。

To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the hash of the stream. When the replication factor is larger than 1, the next subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result.

为了进行哈希查找，distributor找到值大于流哈希的最小适合令牌。当replication factor大于1时，属于不同ingesters的下一个后续令牌（顺时针方向）也将包含在结果中。

The effect of this hash set up is that each token that an ingester owns is responsible for a range of hashes. If there are three tokens with values 0, 25, and 50, then a hash of 3 would be given to the ingester that owns the token 25; the ingester owning token 25 is responsible for the hash range of 1-25.

这种哈希集合的构建效果是：每个哈希值由一个特定的token负责。如果有三个token，它们的值分别是0、25、50，那么哈希值3将被分配给拥有令牌25的ingester；拥有 token 25 的 ingester 负责哈希值范围为 1-25。

Quorum consistency

Since all distributors share access to the same hash ring, write requests can be sent to any distributor.

由于所有distributors共享同一个哈希环，因此可以将写请求发送给任何distributors。

To ensure consistent query results, Loki uses Dynamo-style quorum consistency on reads and writes. This means that the distributor will wait for a positive response of at least one half plus one of the ingesters to send the sample to before responding to the client that initiated the send.

为了确保查询结果的一致性，Loki在读取和写入时使用Dynamo风格的多数一致性。这意味着distributor在向发起发送请求的客户端响应之前，会等待至少51%的ingesters给出正面的响应，才能将样本发送出去。

Ingester

The ingester service is responsible for writing log data to long-term storage backends (DynamoDB, S3, Cassandra, etc.) on the write path and returning log data for in-memory queries on the read path.

ingester服务负责将日志数据写入长期存储后端（例如DynamoDB、S3、Cassandra等），并在读取路径上返回内存查询所需的日志数据。

Ingesters contain a lifecycler which manages the lifecycle of an ingester in the hash ring. Each ingester has a state of either PENDING, JOINING, ACTIVE, LEAVING, or UNHEALTHY:

Ingesters包含一个生命周期管理器lifecycler，用于管理哈希环中Ingesters的生命周期。每个Ingester都有一个状态，可以是PENDING、JOINING、ACTIVE、LEAVING或UNHEALTHY：

Deprecated: the WAL (write ahead log) supersedes this feature

已弃用：WAL（写前日志）已取代此功能

1. PENDING is an Ingester’s state when it is waiting for a handoff from another ingester that is LEAVING.
2. JOINING is an Ingester’s state when it is currently inserting its tokens into the ring and initializing itself. It may receive write requests for tokens it owns.
3. ACTIVE is an Ingester’s state when it is fully initialized. It may receive both write and read requests for tokens it owns.
4. LEAVING is an Ingester’s state when it is shutting down. It may receive read requests for data it still has in memory.
5. UNHEALTHY is an Ingester’s state when it has failed to heartbeat to Consul. UNHEALTHY is set by the distributor when it periodically checks the ring.

Each log stream that an ingester receives is built up into a set of many “chunks” in memory and flushed to the backing storage backend at a configurable interval.

每个ingester接收的日志流在内存中构建成一组“chunks”，并在可配置的时间间隔内刷新到后端存储。

Chunks are compressed and marked as read-only when:

chunks在以下情况下会被压缩并标记为只读：

1. The current chunk has reached capacity (a configurable value).
2. Too much time has passed without the current chunk being updated
3. A flush occurs.

1. 当前块已达到容量（可配置的值）
2. 系统或程序长时间没有更新某个块
3. 发生刷新。

Whenever a chunk is compressed and marked as read-only, a writable chunk takes its place.

If an ingester process crashes or exits abruptly, all the data that has not yet been flushed will be lost. Loki is usually configured to replicate multiple replicas (usually 3) of each log to mitigate this risk.

每当一个chunks被压缩并标记为只读时，一个可写的块就会取而代之。

如果一个 ingester 进程崩溃或突然退出，所有尚未刷新的数据将会丢失。Loki 通常配置为每个日志复制多个副本（通常是3个），以降低这种风险。

When a flush occurs to a persistent storage provider, the chunk is hashed based on its tenant, labels, and contents. This means that multiple ingesters with the same copy of data will not write the same data to the backing store twice, but if any write failed to one of the replicas, multiple differing chunk objects will be created in the backing store. See Querier for how data is deduplicated.

当一个持久性存储提供程序（persistent storage provider）发生刷新时，chunks将根据其租户、标签和内容进行哈希。这意味着具有相同数据副本的多个ingesters不会将相同的数据两次写入后端存储（由于每个块都基于其内容进行了哈希，因此即使多个ingesters有相同的数据副本，它们也不会将相同的数据写入存储两次，因为哈希值会相同，从而识别出重复的数据。）。但是，如果任何写入副本失败，则在后端存储中创建多个不同的chunks对象。请参阅查询器以了解如何进行数据去重。

Timestamp Ordering

Loki is configured to accept out-of-order writes by default.

默认情况下，Loki被配置为接受无序写入。

When not configured to accept out-of-order writes, the ingester validates that ingested log lines are in order. When an ingester receives a log line that doesn’t follow the expected order, the line is rejected and an error is returned to the user.

如果不配置为接受乱序写入，ingester会验证输入的日志行是按顺序的。当ingester接收到不遵循预期顺序的日志行时，该行将被拒绝，并向用户返回一个错误。

The ingester validates that log lines are received in timestamp-ascending order. Each log has a timestamp that occurs at a later time than the log before it. When the ingester receives a log that does not follow this order, the log line is rejected and an error is returned.

数据采集器验证日志行是否按照时间戳升序接收。每条日志的时间戳都比之前的日志发生的时间要晚。

Logs from each unique set of labels are built up into “chunks” in memory and then flushed to the backing storage backend.

If an ingester process crashes or exits abruptly, all the data that has not yet been flushed could be lost. Loki is usually configured with a Write Ahead Log which can be replayed on restart as well as with a replication_factor (usually 3) of each log to mitigate this risk.

从每组独特的标签中获取的日志在内存中构建成“chunks”，然后刷新到后端存储。

如果一个摄取进程崩溃或突然退出，所有尚未刷新的数据都可能丢失。Loki通常配置有预写式日志，在重启时可以replay，同时每个日志的复制因子(通常为3)也可以降低这种风险。

When not configured to accept out-of-order writes, all lines pushed to Loki for a given stream (unique combination of labels) must have a newer timestamp than the line received before it. There are, however, two cases for handling logs for the same stream with identical nanosecond timestamps:

如果没有配置为接受乱序写入，对于给定流（唯一标签组合）的所有推送到Loki的行，其时间戳必须比之前接收到的行的时间戳要新。然而，对于具有相同纳秒时间戳的相同日志流，有两种处理情况:

1. If the incoming line exactly matches the previously received line (matching both the previous timestamp and log text), the incoming line will be treated as an exact duplicate and ignored.
2. If the incoming line has the same timestamp as the previous line but different content, the log line is accepted. This means it is possible to have two different log lines for the same timestamp.

1. 如果输入的行与之前接收的行完全匹配（包括之前的timestamp和日志文本），则将输入的行视为完全重复并忽略。

2. 如果输入的行与上一行的timestamp相同但内容不同，则接受该日志行。这意味着对于相同的timestamp，可能存在两个不同的日志行。

Handoff - Deprecated in favor of the WAL

By default, when an ingester is shutting down and tries to leave the hash ring, it will wait to see if a new ingester tries to enter before flushing and will try to initiate a handoff. The handoff will transfer all of the tokens and in-memory chunks owned by the leaving ingester to the new ingester.

默认情况下，当一个 Ingester 正在关闭并试图离开哈希环时，它会等待看是否有新的 Ingester 在刷新之前试图进入，并尝试发起一次移交。移交将把离开的 Ingester 拥有的所有令牌和内存块转移给新的 Ingester。

Before joining the hash ring, ingesters will wait in PENDING state for a handoff to occur. After a configurable timeout, ingesters in the PENDING state that have not received a transfer will join the ring normally, inserting a new set of tokens.

This process is used to avoid flushing all chunks when shutting down, which is a slow process.

在加入哈希环之前，ingesters将处于 PENDING 状态等待交接。在达到可配置的超时时间后，处于 PENDING 状态且未收到转移的ingesters将正常加入环，并插入一组新令牌。

这个过程用于在关闭时避免刷新所有chunks，这是一个缓慢的过程。

Filesystem Support

While ingesters do support writing to the filesystem through BoltDB, this only works in single-process mode as queriers need access to the same back-end store and BoltDB only allows one process to have a lock on the DB at a given time.

虽然摄取程序确实支持通过BoltDB写入文件系统，但这只适用于单进程模式，因为查询程序需要访问同一后端存储，而BoltDB在给定时间只允许一个进程锁定数据库。

Query Frontend

The query frontend is an optional service providing the querier’s API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers. The querier service will be still required within the cluster, in order to execute the actual queries.

查询前端query frontend是一个可选服务，提供查询器的API端点，可用于加速读取路径。

当query frontend就位时，传入的查询请求应定向到query frontend，而不是queriers.。集群中仍然需要查询器服务，以便执行实际查询。

The query frontend internally performs some query adjustments and holds queries in an internal queue. In this setup, queriers act as workers which pull jobs from the queue, execute them, and return them to the query-frontend for aggregation. Queriers need to be configured with the query frontend address (via the -querier.frontend-address CLI flag) in order to allow them to connect to the query frontends.

query frontend在内部执行一些查询调整，并将查询保留在内部队列中。在这个设置中，queriers充当workers，从队列中拉取作业，执行它们，然后将它们返回给query frontend进行聚合。为了允许queriers连接到query frontend，需要使用-querier.frontend-address CLI标志配置queriers的query frontend地址。

Query frontends are stateless. However, due to how the internal queue works, it’s recommended to run a few query frontend replicas to reap the benefit of fair scheduling. Two replicas should suffice in most cases.

Query frontends是无状态的。但是，由于内部队列的工作方式，建议运行一些Query frontends副本，以获得公平调度的好处。在大多数情况下，两个副本就足够了。

Queueing

The query frontend queuing mechanism is used to:

查询前端排队机制用于：

• Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the TCO.
• Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO).
• Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants.

• 确保在失败时重试可能导致查询器内存不足（OOM）错误的大型查询。这允许管理员为查询提供不足的内存，或者乐观地并行运行更多的小型查询，这有助于降低TCO。
• 通过使用先进先出队列（FIFO）将多个大型请求分布在所有查询器中，防止它们在单个查询器上被护送。
• 通过公平地安排租户之间的查询，防止单个租户拒绝其他租户的服务。

Splitting

The query frontend splits larger queries into multiple smaller queries, executing these queries in parallel on downstream queriers and stitching the results back together again. This prevents large (multi-day, etc) queries from causing out of memory issues in a single querier and helps to execute them faster.

查询前端将较大的查询拆分为多个较小的查询，在下游查询器上并行执行这些查询，并将结果重新缝合在一起。这可以防止大型（多日等）查询在单个查询器中导致内存不足问题，并有助于更快地执行这些查询。

Caching

Metric Querires

The query frontend supports caching metric query results and reuses them on subsequent queries. If the cached results are incomplete, the query frontend calculates the required subqueries and executes them in parallel on downstream queriers. The query frontend can optionally align queries with their step parameter to improve the cacheability of the query results. The result cache is compatible with any loki caching backend (currently memcached, redis, and an in-memory cache).

查询前端支持缓存度量查询结果，并在后续查询中重用它们。如果缓存的结果不完整，则查询前端计算所需的子查询，并在下游查询器上并行执行这些子查询。查询前端可以选择性地将查询与其步骤参数对齐，以提高查询结果的可缓存性。结果缓存与任何loki缓存后端兼容（目前为memcached、redis和内存中缓存）。

Log Querirs - Comming soon!

Caching log (filter, regexp) queries are under active development.

缓存日志（filter、regexp）查询正在积极开发中。

Querier

The querier service handles queries using the LogQL query language, fetching logs both from the ingesters and from long-term storage.

查询器服务使用LogQL查询语言处理查询，从摄取器和长期存储中获取日志。

Queriers query all ingesters for in-memory data before falling back to running the same query against the backend store. Because of the replication factor, it is possible that the querier may receive duplicate data. To resolve this, the querier internally deduplicates data that has the same nanosecond timestamp, label set, and log message.

查询器在返回到对后端存储运行相同的查询之前，先向所有摄取器查询内存中的数据。由于复制因素，查询器可能会接收到重复的数据。为了解决此问题，查询器在内部消除具有相同纳秒时间戳、标签集和日志消息的重复数据。

At read path, replication factor also plays a role here. For example with replication-factor of 3, we require that two queries to be running.

在读取路径上，复制因子也在这里发挥作用。例如，在复制因子为3的情况下，我们要求运行两个查询。

Labels 标签

Labels are key value pairs and can be defined as anything! We like to refer to them as metadata to describe a log stream. If you are familiar with Prometheus, there are a few labels you are used to seeing like job and instance, and I will use those in the coming examples.

The scrape configs we provide with Grafana Loki define these labels, too. If you are using Prometheus, having consistent labels between Loki and Prometheus is one of Loki’s superpowers, making it incredibly easy to correlate your application metrics with your log data.

标签是键值对，可以定义为任何东西！我们喜欢把它们称为元数据，用来描述一个日志流。如果你熟悉Prometheus，你会习惯看到一些标签，比如job和instance，在接下来的例子中，我会使用这些标签。

• 这里提到了Prometheus，一个开源的监控和告警工具，它经常使用像“job”和“instance”这样的标签。“job”可能描述一个特定任务或服务的名称，而“instance”可能标识一个运行中的实例。

我们为Grafana Loki提供的配置中，也定义了这些标签。如果你在使用Prometheus，在Loki和Prometheus之间保持标签一致是Loki的一个超级功能，可以非常轻松地将你的应用程序指标与日志数据关联起来。

How Loki uses labels Loki怎样使用标签

Labels in Loki perform a very important task: They define a stream. More specifically, the combination of every label key and value defines the stream. If just one label value changes, this creates a new stream.

If you are familiar with Prometheus, the term used there is series; however, Prometheus has an additional dimension: metric name. Loki simplifies this in that there are no metric names, just labels, and we decided to use streams instead of series.

在Loki中，标签执行了一项非常重要的任务：它们定义了一个数据流。更具体地说，每个标签键和值的组合定义了一个数据流。如果只是一个标签的值发生了变化，那么就会创建一个新的数据流。如果你熟悉Prometheus，Prometheus中使用的术语是“系列”（series）。然而，Prometheus还有一个额外的维度：度量名称（metric name）。与Prometheus相比，Loki对此进行了简化，因为它没有度量名称，只有标签。我们决定使用“流”（streams）而不是“系列”（series）。

Note: Structured metadata do not define a stream, but are metadata attached to a log line. See structured metadata for more information.

注意：结构化元数据并不定义一个流，而是附加到日志行的元数据。有关更多信息，请参阅结构化元数据。

Format 格式

Loki places the same restrictions on label naming as Prometheus:

Loki 对标签命名施加与 Prometheus 相同的限制：

It may contain ASCII letters and digits, as well as underscores and colons. It must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*.

Note: The colons are reserved for user defined recording rules. They should not be used by exporters or direct instrumentation.

它可以包含ASCII字母和数字，以及下划线和冒号。它必须匹配正则表达式[a-zA-Z_:][a-zA-Z0-9_:]*。

注意：冒号是保留给用户定义的记录规则的。导出器或直接仪器不应使用它们。

Loki labels demo

This series of examples will illustrate basic use cases and concepts for labeling in Loki.

Let’s take an example:

这一系列实例将阐明Loki中label的基本使用方法和概念

例子：

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: syslog
      __path__: /var/log/syslog

This config will tail one file and assign one label: job=syslog. This will create one stream in Loki.

You could query it like this:

此配置将跟踪一个文件并分配一个Label：job = syslog。这将在Loki中创建一个流。

可以这样查询：

{job="syslog"}

扩展上面的例子：

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: syslog
      __path__: /var/log/syslog
 - job_name: apache
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: apache
      __path__: /var/log/apache.log

Now we are tailing two files. Each file gets just one label with one value, so Loki will now be storing two streams.

We can query these streams in a few ways:

现在我们正在跟踪两个文件，每个文件只有一个Label和一个值，因此Loki现在将存储两个流。

我们可以通过几种方式查询这些流：

{job="apache"} <- show me logs where the job label is apache
{job="syslog"} <- show me logs where the job label is syslog
{job=~"apache|syslog"} <- show me logs where the job is apache **OR** syslog

In that last example, we used a regex label matcher to view log streams that use the job label with one of two possible values. Now consider how an additional label could also be used:

最后一个例子，使用正则表达式标签匹配器查看使用具有两个可能值之一的作业标签的日志流。现在考虑如何使用额外的标签：

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: syslog
      env: dev
      __path__: /var/log/syslog
 - job_name: apache
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: apache
      env: dev
      __path__: /var/log/apache.log

除了正则表达式，还可以这样做：

{env="dev"} <- will return all logs with env=dev, in this case this includes both log streams

Hopefully now you are starting to see the power of labels. By using a single label, you can query many streams. By combining several different labels, you can create very flexible log queries.

Labels are the index to Loki’s log data. They are used to find the compressed log content, which is stored separately as chunks. Every unique combination of label and values defines a stream, and logs for a stream are batched up, compressed, and stored as chunks.

For Loki to be efficient and cost-effective, we have to use labels responsibly. The next section will explore this in more detail.

现在您开始看到标签的作用了。通过使用单个标签，您可以查询多个流。通过组合多个不同的标签，您可以创建非常灵活的日志查询。

标签是 Loki 日志数据的索引。它们用于查找已压缩的日志内容，这些内容作为chunks单独存储。标签和值的唯一组合定义了一个流，而流的日志被批处理、压缩并存储为chunks。

为了使 Loki 高效且经济高效，我们必须负责任地使用标签。下一部分将更详细地探讨这一点。

cardinality 基数

The two previous examples use statically defined labels with a single value; however, there are ways to dynamically define labels. Let’s take a look using the Apache log and a massive regex you could use to parse such a log line:

前面的两个例子使用静态定义的标签和单个值；然而，也有一些方法可以动态定义标签。让我们使用Apache日志和一个巨大的正则表达式来分析这样的日志行：

nohighlight Copy

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

yaml Copy

- job_name: system
   pipeline_stages:
      - regex:
        expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status_code>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"]*)\"?\\s?\"?(?P<useragent>[^\"]*)?\"?$"
    - labels:
        action:
        status_code:
   static_configs:
   - targets:
      - localhost
     labels:
      job: apache
      env: dev
      __path__: /var/log/apache.log

This regex matches every component of the log line and extracts the value of each component into a capture group. Inside the pipeline code, this data is placed in a temporary data structure that allows using it for several purposes during the processing of that log line (at which point that temp data is discarded). Much more detail about this can be found in the Promtail pipelines documentation.

这个正则表达式匹配了一行日志的的每个组成部分，并将每个部分的值提取到捕获组中。在管道代码中，将这个数据放到一个临时结构中，以便在处理该日志行时将其用于多种目的。（此时将丢弃该临时数据）。有关此更多详细信息参阅文档。。

From that regex, we will be using two of the capture groups to dynamically set two labels based on content from the log line itself:

从这个正则表达式中，我们使用两个捕获组根据日志行本身的动态内容动态地设置两个标签：

action (for example, action="GET", action="POST")

status_code (for example, status_code="200", status_code="400")

And now let’s walk through a few example lines:

现在让我们通过几个实例进行了解：

nohighlight Copy

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

In Loki the following streams would be created:

在Loki中以下的流将被创建：

Copy

{job="apache",env="dev",action="GET",status_code="200"} 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="200"} 11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="GET",status_code="400"} 11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="400"} 11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

Those four log lines would become four separate streams and start filling four separate chunks.

这四条日志将变成四个独立的流，并开始填充四个独立的chunks

Any additional log lines that match those combinations of label/values would be added to the existing stream. If another unique combination of labels comes in (for example, status_code="500") another new stream is created.

任何与这些Label/值组合匹配的额外日志行将被添加到现有的流中。如果标签的另一个唯一组合出现，（例如，status_code=“500”），则将创建另一个新流。

Imagine now if you set a label for ip. Not only does every request from a user become a unique stream. Every request with a different action or status_code from the same user will get its own stream.

现在想象一下，如果为IP设置了标签，那么不仅来自同一用户的每个请求都成为唯一的流，来自同一用户的具有不同操作或状态代码的每个请求都将获得自己的流。

Doing some quick math, if there are maybe four common actions (GET, PUT, POST, DELETE) and maybe four common status codes (although there could be more than four!), this would be 16 streams and 16 separate chunks. Now multiply this by every user if we use a label for ip. You can quickly have thousands or tens of thousands of streams.

做一个简要的计算：如果常见的HTTP动作和常见的HTTP状态码每种对应一条流和一块数据，那么我们为每个用户打上IP标签的话，就可以快速创建成千上万甚至数十万的流。

This is high cardinality, and it can lead to significant performance degredation.

这是高基数（high cardinality）的，可能会导致显著的性能下降。

When we talk about cardinality we are referring to the combination of labels and values and the number of streams they create. High cardinality is using labels with a large range of possible values, such as ip, or combining many labels, even if they have a small and finite set of values, such as using status_code and action.

当我们谈论cardinality时，我们指的是标签和值的组合以及它们创建的流的数量。高基数是使用具有大量可能值得标签，例如IP，或者组合许多标签，即使他们具有小而有限得值的集合，例如使用状态代码的动作。

High cardinality causes Loki to build a huge index and to flush thousands of tiny chunks to the object store. Loki currently performs very poorly in this configuration. If not accounted for, high cardinality will significantly reduce the operability and cost-effectiveness of Loki.

高基数会导致Loki构建一个巨大的索引，并将数以千计的小块数据写入对象存储。在这种情况下，Loki的性能表现非常差。如果未加处理，高基数将显著降低Loki的可操作性和成本效益。

Optimal Loki performance with parallelization 通过并行化实现最优的 Loki 性能

Now you may be asking: If using too many labels—or using labels with too many values—is bad, then how am I supposed to query my logs? If none of the data is indexed, won’t queries be really slow?

现在你可能会问：如果使用过多标签或使用具有太多值的标签是件坏事，那我该如何查询我的日志呢？如果数据没有被索引，查询速度会不会很慢？

As we see people using Loki who are accustomed to other index-heavy solutions, it seems like they feel obligated to define a lot of labels in order to query their logs effectively. After all, many other logging solutions are all about the index, and this is the common way of thinking.

我们发现，习惯于使用其他索引密集型解决方案的人在使用Loki时，似乎觉得有必要定义大量标签才能有效地查询他们的日志。毕竟，许多其他日志记录解决方案都是关于索引的，这是常见的思维方式。

When using Loki, you may need to forget what you know and look to see how the problem can be solved differently with parallelization. Loki’s superpower is breaking up queries into small pieces and dispatching them in parallel so that you can query huge amounts of log data in small amounts of time.

当使用Loki时，你可能需要忘掉你已知的东西，并尝试用并行化的方式去解决这个问题。Loki的超级能力在于能够将查询分解成 小部分，并将它们并行分派，从而使得你能够在短时间内查询大量日志数据。

This kind of brute force approach might not sound ideal, but let me explain why it is.

这种暴力的方法可能听起来并不理想，但让我解释一下为什么它是有效的。

Large indexes are complicated and expensive. Often a full-text index of your log data is the same size or bigger than the log data itself. To query your log data, you need this index loaded, and for performance, it should probably be in memory. This is difficult to scale, and as you ingest more logs, your index gets larger quickly.

大型索引复杂而且昂贵。通常，日志数据的全文索引和日志数据本身的大小相同或更大。每当要查询日志数据时都要加载索引，为了查询性能，应该把索引存储在内存中。这导致（索引？）很难扩展，并且随着日志量的增加，索引会迅速增大。

Now let’s talk about Loki, where the index is typically an order of magnitude smaller than your ingested log volume. So if you are doing a good job of keeping your streams and stream churn to a minimum, the index grows very slowly compared to the ingested logs.

现在让我门来谈谈Loki。Loki的索引大小通常比收集到的日志小一个数量级。因此，如果你能很好的保持你的流和流消耗最小化，与摄入的日志相比，索引的增长速度非常缓慢。

Loki will effectively keep your static costs as low as possible (index size and memory requirements as well as static log storage) and make the query performance something you can control at runtime with horizontal scaling.

Loki有效地将静态成本降到最低（索引大小、内存要求以及静态日志存储），并通过水平扩展使查询性能在运行时可控。

To see how this works, let’s look back at our example of querying your access log data for a specific IP address. We don’t want to use a label to store the IP address. Instead we use a filter expression to query for it:

要了解Loki是怎么工作的，回顾查询访问日志数据中特定IP地址的示例。这次，不使用标签来存储IP地址，而是使用过滤表达式来查询它：

Copy

{job="apache"} |= "11.11.11.11"

Behind the scenes, Loki will break up that query into smaller pieces (shards), and open up each chunk for the streams matched by the labels and start looking for this IP address.

在后台，Loki会将该查询分解成更小的部分（分片），并为匹配Label的流打开每个chunk，然后开始查找这个IP地址。

The size of those shards and the amount of parallelization is configurable and based on the resources you provision. If you want to, you can configure the shard interval down to 5m, deploy 20 queriers, and process gigabytes of logs in seconds. Or you can go crazy and provision 200 queriers and process terabytes of logs!

这些分片的规模和并行化的数量是可以配置的，并且基于你提供的资源。如果愿意，可以将分片间隔配置为5m，部署 20个查询器，并在几秒钟之内处理千兆级别（gigabytes）的日志。或者您也可以提供200个查询器，处理数太字节（terabytes）的日志。

This trade-off of smaller index and parallel brute force querying vs. a larger/faster full-text index is what allows Loki to save on costs versus other systems. The cost and complexity of operating a large index is high and is typically fixed – you pay for it 24 hours a day if you are querying it or not.

这种小索引和并行暴力查询与更大/更快的全文索引之间的权衡，使得Loki相对于其它系统能够更节省成本。大型索引的操作成本和复杂性都很高，并且通常是固定的——无论是否进行查询，都需要为其支付24小时的费用。

The benefits of this design mean you can make the decision about how much query power you want to have, and you can change that on demand. Query performance becomes a function of how much money you want to spend on it. Meanwhile, the data is heavily compressed and stored in low-cost object stores like S3 and GCS. This drives the fixed operating costs to a minimum while still allowing for incredibly fast query capability.

这种设计的好处在于，您可以根据需要自行决定想要拥有多少查询能力，并可随时进行更改。查询性能将取决于您愿意为此投入多少资金。同时，数据将经过高度压缩并存储在像S3和GCS这样的低成本对象存储中。这使得固定运营成本降至最低，同时仍然能保持令人难以置信的快速查询能力。

Label best practices 最佳Label实践

Grafana Loki 正在积极开发中，我们也在不断努力提高性能。但以下是一些最新的标签最佳实践，将为您提供最佳的 Loki 体验。

Static labels are good 静态标签为佳

Things like, host, application, and environment are great labels. They will be fixed for a given system/app and have bounded values. Use static labels to make it easier to query your logs in a logical sense (e.g. show me all the logs for a given application and specific environment, or show me all the logs for all the apps on a specific host).

像主机、应用程序和环境这样的标签都是好的标签。它们在给定的系统/应用程序中是固定的且有界的。使用静态标签可以更轻松的按逻辑查询日志（例如，显示给定应用程序和特定环境的所有日志，或者显示特定主机上所有应用程序的所有日志）

Use dynamic labels sparingly 谨慎使用动态标签

Too many label value combinations leads to too many streams. The penalties for that in Loki are a large index and small chunks in the store, which in turn can actually reduce performance.

过多的标签会导致流过多。在Loki中，这种情况带来的负面影响是产生大量索引和小存储chunks。这可能会降低Loki的性能。

To avoid those issues, don’t add a label for something until you know you need it! Use filter expressions (|= "text", |~ "regex", …) and brute force those logs. It works – and it’s fast.

为了避免这些问题，在拥有确切需求之前不要随意添加标签。使用过滤表达式和暴力方法处理日志。这样在保证查询有效的前提下速度快。

If you often parse a label from a log line at query time, the label has a high cardinality, and extracting that label is expensive in terms of performance; consider extracting the label on the client side attaching it as structured metadata to log lines .

？如果你在查询时经常从日志行中解析标签，那么该标签具有高基数，并且提取该标签在性能方面开销很大；考虑在客户端提取标签并将其作为structured metadata附加到日志行中。

From early on, we have set a label dynamically using Promtail pipelines for level. This seemed intuitive for us as we often wanted to only show logs for level="error"; however, we are re-evaluating this now as writing a query. {app="loki"} |= "level=error" is proving to be just as fast for many of our applications as {app="loki",level="error"}.

早先，我们使用Promtail管道为Level动态设置了一个标签。这样做的目的很直观，因为我们经常只想显示Level = "error"的日志；然而，我们现在正在重新审视这个做法。因为在Loki中编写查询程序{app="loki"} |= "level=error"和使用{app="loki",level="error"}一样块。

This may seem surprising, but if applications have medium to low volume, that label causes one application’s logs to be split into up to five streams, which means 5x chunks being stored. And loading chunks has an overhead associated with it. Imagine now if that query were {app="loki",level!="debug"}. That would have to load way more chunks than {app="loki"} != "level=debug".

这听起来是令人惊讶的，但是如果应用程序的日志量适中或者较小，该标签会导致一个应用程序的日志被分割成最多5个流，这意味着要存储5倍的数据块。加载数据快快还需要一些开销。现在想象一下，若查询为{app="loki",level!="debug"}，那么这个查询将加载比使用{app="loki"} != "level=debug"查询更多的chunks。

Above, we mentioned not to add labels until you need them, so when would you need labels?? A little farther down is a section on chunk_target_size. If you set this to 1MB (which is reasonable), this will try to cut chunks at 1MB compressed size, which is about 5MB-ish of uncompressed logs (might be as much as 10MB depending on compression). If your logs have sufficient volume to write 5MB in less time than max_chunk_age, or many chunks in that timeframe, you might want to consider splitting it into separate streams with a dynamic label.

那么，什么时候需要添加标签呢？下面我们将讨论关于chunk_target_size的部分。如果你将其设置为1MB（这是合理的），那么它将以1MB的压缩大小来切割块，这大约对应于5MB的未压缩的日志（根据压缩情况，可能高达10MB）。如果你的日志量足够大，使得在max_chunk_age时间内可以写入5MB，或者在该时间段内产生了很多chunks的数据，那么你可能需要考虑使用动态标签将其分割成单独的流。

What you want to avoid is splitting a lg file into streams, which result in chunks getting flushed because the stream is idle or hits the max age before being full. As of Loki 1.4.0, there is a metric which can help you understand why chunks are flushed sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m])).

你应当避免：将一个日志文件分割成多个流（streams）后导致块（chunks）被s刷新（flushed）掉。这些块被刷新的原因可能是流处于空闲状态，或者在满之前达到了最大年龄。

从Loki 1.4.0版本开始，有一个度量标准sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m])).可以帮助你理解为什么块会被刷新。

It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a level dynamic label, just |= "level=debug" instead.

当某些数据块（chunk）被刷新时，不必要求每个数据块都必须是满的。但这（刷新前填满chunk）确实会对操作的许多方面有所改善。因此，我们目前的建议是尽量避免使用动态标签，而是倾向于使用过滤表达式。例如，不要添加一个动态标签level，而使用“|=”操作符和“level=debug”这样的表达式。

Label values must always be bounded 保证Label取值有界

If you are dynamically setting labels, never use a label which can have unbounded or infinite values. This will always result in big problems for Loki.

如果正在使用动态标签，要保证标签的取值范围有界。否则会导致严重问题。

Try to keep values bounded to as small a set as possible. We don’t have perfect guidance as to what Loki can handle, but think single digits, or maybe 10’s of values for a dynamic label. This is less critical for static labels. For example, if you have 1,000 hosts in your environment it’s going to be just fine to have a host label with 1,000 values.

尽量保持label值限定在尽可能小的集合中。我们对于Loki能处理的内容没有完美的指导，但可以考虑单个数字，或者对于动态标签来说，考虑10个值左右。对于静态标签来说，这个问题并不那么关键或重要。例如，如果你在一个环境中有一千台主机，那么有一个包含一千个值的主机标签是完全可以的。

Be aware of dynamic labels applied by clients 注意客户端使用的动态标签

Loki has several client options: Promtail (which also supports systemd journal ingestion and TCP-based syslog ingestion), Fluentd, Fluent Bit, a Docker plugin, and more!

Loki 有多种客户端选项：Promtail（也支持 systemd 日志摄取和基于 TCP 的 syslog 摄取）、Fluentd、Fluent Bit、Docker 插件等等！

Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied. Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality. Series information can be queried through the Series API, or you can use logcli.

每种客户端都有配置日志流标签的方法。但要注意，何种动态标签可能会被使用。建议使用Loki系列API来了解你的日志流是什么样子的，并查看是否有可能减少流的数量和基数。可以通过 Series API查询系列信息，或者您可以使用logcli。

In Loki 1.6.0 and newer the logcli series command added the --analyze-labels flag specifically for debugging high cardinality labels:

在Loki 1.6.0 及更高版本中，logcli系列命令添加了--analyze-labels标志，专门用于调试高基数标签。

Copy

Total Streams:  25017
Unique Labels:  8

Label Name  Unique Values  Found In Streams
requestId   24653          24979
logStream   1194           25016
logGroup    140            25016
accountId   13             25016
logger      1              25017
source      1              25016
transport   1              25017
format      1              25017

In this example you can see the requestId label had a 24653 different values out of 24979 streams it was found in, this is bad!!

在这个例子中，可以看到requestId标签在24979个流中找到，其中具有24653个不同的值。这是很糟糕的！！（对查询性能而言）

This is a perfect example of something which should not be a label, requestId should be removed as a label and instead filter expressions should be used to query logs for a specific requestId. For example if requestId is found in the log line as a key=value pair you could write a query like this: {logGroup="group1"} |= "requestId=32422355"

这是一个很好的例子，它说明了某些东西不应该被当作标签。在这里，requestId不应该作为一个标签存在，而应该使用过滤表达式来查询特定的requestId的日志。例如，如果requestId作为一个key-value对出现在日志行中，你可以这样编写查询： {logGroup="group1"} |= "requestId=32422355"

What is Stuctured metadata 什么是结构化元数据

Warning: Structured metadata is an experimental feature and is subject to change in future releases of Grafana Loki. This feature is not yet available for Cloud Logs users.

警告：结构化元数据是一个实验性功能，在Grafana Loki的未来版本中可能会有所更改。此功能尚不适用于Cloud Logs用户。

Warning: Structured metadata was added to chunk format V4 which is used if the schema version is greater or equal to 13. (See Schema Config for more details about schema versions. )

警告：如果模式版本大于或等于13，则将结构化元数据添加到块格式V4中。 (有关模式版本的更多详细信息，请参见模式配置。 )

One of the powerful features of Loki is parsing logs at query time to extract metadata and build labels out of it. However, the parsing of logs at query time comes with a cost which can be significantly high for, as an example, large json blobs or a poorly written query using complex regex patterns.

Loki 的强大功能之一是在查询时解析日志以提取元数据并从中构建标签。然而，在查询时解析日志是有代价的，当日志是大量的 JSON 数据时，或使用复杂的正则表达式模式编写查询时，其解析代价可能非常高。

In addition, the data extracted from logs at query time is usually high cardinality, which can’t be stored in the index as it would increase the cardinality too much, and therefore reduce the performance of the index.

另外，在查询时从日志中提取的数据通常具有高基数，不能存储在索引中，因为这会增加基数的数量，从而降低索引的性能。

Structured metadata is a way to attach metadata to logs without indexing them. Examples of useful metadata are trace IDs, user IDs, and any other label that is often used in queries but has high cardinality and is expensive to extract at query time.

结构化元数据（Structured metadata）是一种将元数据附加到日志而不需要对它们进行索引的方法。有用的元数据示例包括追踪ID（trace IDs）、用户ID（user IDs）以及在查询中经常使用但具有高基数且在查询时提取成本较高的其他标签。

Attaching structured metadata to log lines 为日志行附加结构化元数据

You have the option to attach structured metadata to log lines in the push payload along with each log line and the timestamp. For more information on how to push logs to Loki via the HTTP endpoint, refer to the HTTP API documentation.

当你推送日志数据时，你可以选择为每条日志行附加结构化的元数据。这些元数据可以与日志行和时间戳一起发送。结构化的元数据意味着它有一个预定义的模式或结构，这使得它更容易被机器解析和理解。如果你想了解更多关于如何通过HTTP端点将日志推送到Loki的详细信息，你应该参考HTTP API的文档。这提供了具体指导和技术细节，告诉你如何利用HTTP API将日志发送到Loki系统。

Alternatively, you can use the Grafana Agent or Promtail to extract and attach structured metadata to your log lines. See the Promtail: Structured metadata stage for more information.

除了其他方法之外，用户还可以选择使用Grafana Agent或Promtail来从日志行中提取和附加结构化的元数据。建议用户查看Promtail的“Structured metadata stage”部分以获取更多详细信息。

Querying structured metadata 查询结构化元数据

Structured metadata is extracted automatically for each returned log line and added to the labels returned for the query. You can use labels of structured metadata to filter log line using a label filter expression.

对于每个返回的日志行，Loki会自动提取结构化元数据，

• 结构化元数据是指以特定格式存储的数据，例如JSON、XML等。这种格式的数据更容易被机器解析和处理。

并将其添加到查询返回的标签中。您可以使用结构化元数据的标签来使用标签过滤表达式过滤日志行。

For example, if you have a label trace_id attached to some of your log lines as structured metadata, you can filter log lines using:

例如，如果您将一些日志行附加了trace_id标签作为结构化元数据，则可以使用以下内容过滤日志行：

logqlCopy

{job="example"} | trace_id="0242ac120002"`

Of course, you can filter by multiple labels of structured metadata at the same time:

当然，您可以同时通过结构化元数据的多个标签进行筛选：

logqlCopy

{job="example"} | trace_id="0242ac120002" | user_id="superUser123"

Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like maximum of series (50000) reached for a single query. You can use the Keep and Drop stages to filter out labels that you don’t need. For example:

请注意，由于结构化元数据是自动提取到结果标签中的，因此一些度量查询可能会返回errpr，例如maximum of series (50000) reached for a single query。您可以使用“Keep”和“Drop”阶段来过滤掉不需要的标签。例如：

logqlCopy

count_over_time({job="example"} | trace_id="0242ac120002" | keep job  [5m])

Consistent hash rings 一致哈希环

Consistent hash rings are incorporated into Loki cluster architectures to

• aid in the sharding of log lines
• implement high availability
• ease the horizontal scale up and scale down of clusters. There is less of a performance hit for operations that must rebalance data.

一致性哈希环被集成到Loki集群架构中，以：

• 辅助日志行的分片
• 实现高可用性
• 简化集群的水平扩展和缩减。对于必须重新平衡数据的操作，性能影响较小。

Hash rings connect instances of a single type of component when

• there are a set of Loki instances in monolithic deployment mode
• there are multiple read components or multiple write components in simple scalable deployment mode
• there are multiple instances of one type of component in microservices mode

在以下情形，哈希环连接同一类型组件的实例：

• 在单体部署模式下有一组 Loki 实例
• 在简单可扩展部署模式下有一个或多个读取组件或多个写入组件
• 在微服务模式下有一个组件类型的多个实例

Not all Loki components are connected by hash rings. These components need to be connected into a hash ring:

• distributors
• ingesters
• query schedulers
• compactors
• rulers

并非所有的Loki组件都通过哈希环连接。这些组件需要被连接成一个哈希环：

• distributors（分发器）
• ingesters（摄取器）
• query schedulers（查询调度器）
• compactors（整理器）
• rulers

These components can optionally be connected into a hash ring:

• index gateway

这些组件可以选择性地连接到哈希环中：

• 索引网关

In an architecture that has three distributors and three ingesters defined, the hash rings for these components connect the instances of same-type components.

在具有三个 distributors和三个ingesters 的架构中，这些组件的哈希环将相同类型组件的实例连接在一起。

Each node in the ring represents an instance of a component. Each node has a key-value store that holds communication information for each of the nodes in that ring. Nodes update the key-value store periodically to keep the contents consistent across all nodes. For each node, the key-value store holds:

• an ID of the component node
• component address, used by other nodes as a communication channel
• an indication of the component node’s health

环中的每个节点代表一个组件实例。每个节点都有一个键值存储器，用于保存该环中每个节点的通信信息。节点会定期更新键值存储器，以保持所有节点内容的一致性。对于每个节点，键值存储器包含：

• 组件节点的ID
• 其他节点用作通信通道的组件地址
• 组件节点健康状况的指示

Configuring rings 一致环

Define ring configuration within the common.ring_config block.

在common.ring_config块中定义环配置。

Use the default memberlist key-value store type unless there is a compelling reason to use a different key-value store type. memberlist uses a gossip protocol to propagate information to all the nodes to guarantee the eventual consistency of the key-value store contents.

除非有充分的理由使用不同的键值存储类型，否则使用默认的成员列表键值存储类型。memberlist使用gossip协议将信息传播到所有节点，以保证键值存储内容的最终一致性。

There are additional configuration options for distributor rings, ingester rings, and ruler rings. These options are for advanced, specialized use only. These options are defined within the distributor.ring block for distributors, the ingester.lifecycler.ring block for ingesters, and the ruler.ring block for rulers.

还有额外的配置选项可用于distributor环、ingester 环和ruler环。这些选项仅适用于高级和特殊用途。这些选项在distributor的distributor.ring块中定义，ingester的ingester.lifecycler.ring块中定义，以及ruler的ruler.ring块中定义。

About the distributor ring

Distributors use the information in their key-value store to keep a count of the quantity of distributors in the distributor ring. The count further informs cluster limits.

Distributors 使用其键值存储中的信息来统计Distributors 环中的Distributors 数量。这个数量进一步用于设定集群的限制。

About the igester ring

Ingester ring information in the key-value stores is used by distributors. The information lets the distributors shard log lines, determining which ingester or set of ingesters a distributor sends log lines to.

在键值存储中，分发器使用Ingester环信息。这些信息使分发器能够对日志行进行分片，确定分发器将日志行发送到哪个Ingester或Ingester集。

About the query scheduler ring

Query schedulers use the information in their key-value store for service discovery of the schedulers. This allows queriers to connect to all available schedulers, and it allows schedulers to connect to all available query frontends, effectively creating a single queue that aids in balancing the query load.

查询调度器使用其键值存储中的信息进行调度器的服务发现。 这允许查询器连接到所有可用的调度器，并允许调度器连接到所有可用的查询前端，从而有效地创建一个有助于平衡查询负载的单个队列。

About the compactor ring

Compactors use the information in the key-value store to identify a single compactor instance that will be responsible for compaction. The compactor is only enabled on the responsible instance, despite the compactor target being on multiple instances.

压缩器使用键值存储中的信息来识别负责进行压缩的单个压缩器实例。尽管压缩器目标位于多个实例上，但只有负责的实例上启用了压缩器。

About the ruler ring

The ruler ring is used to determine which rulers evaluate which rule groups.

ruler ring用于确定哪些 rule评估哪些规则组。

About the index gateway ring

The index gateway ring is used to determine which gateway is responsible for which tenant’s indexes when queried by rulers or queriers.

索引网关环用于确定在由ruler或querier查询时，哪个网关负责哪个租户的索引。