Prometheus远程写Metric Java代码实现

参考地址：https://atchison.dev/prometheus-remote-write-in-java/

gogo.proto文件地址

https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto

remote.proto type.proto文件地址

https://github.com/prometheus/prometheus/tree/master/prompb

大致思路先写下，prometheus.yml加个配置项，远程写是调用http请求来写的，数据格式是protobuf（一种自定义的编码格式），编码格式是snappy（一种压缩格式）；远程写通过snappy先压缩，然后protobuf编码的字节数组请求的；接收到远程写数据时是乱码，先用snappy进行解压缩，prometheus官网文档远程写提供remote.proto（包含编码和解码），remote.proto文件中依赖了types.proto和gogo.proto两个文件，我是在prometheus源码的包里边找到的，所以需要把这三个protobuf文件生成java文件，调用Remote.WriteRequest.parseFrom来解码

一、protobuf转java

1. 下载Protocol Buffers
地址：https://github.com/protocolbuffers/protobuf/releases
Mac解压，如图：

2. 把remote.proto、types.proto、gogo.proto三个文件copy的bin下面 如图：

3. protoc生成java文件

// src_dir: .proto 所在的源目录
// --java_out: 生成 java 代码
// dst_dir: 生成代码的目标目录
// xxx.proto: 要针对哪个 proto 文件生成接口代码
protoc -I=src_dir --java_out=dst_dir src_dir/xxx.proto

如图三个protobuf文件生成三个java文件：

二、prometheus.yml配置

远程写地址

emote_write:
 - url: "http://localhost:8080/receive"
   # 基本认证 这是每个远程写请求的授权头属性
   basic_auth:
     username: prometheus 
     password: prometheus

三、java代码（三个java文件copy到项目中）

package com.jyh.prome;
import com.jyh.prome.protobuf.Remote;
import com.jyh.prome.protobuf.Types;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.xerial.snappy.Snappy;
import javax.servlet.http.HttpServletRequest;
import java.io.*;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

@RestController
public class RemoteWrite {

    @RequestMapping(value = "/receive")
    public void write(@RequestBody byte[] body, HttpServletRequest request) throws IOException, InterruptedException {
        byte[] a = Snappy.uncompress(body);
        List<Types.TimeSeries> timeSeriesList = Remote.WriteRequest.parseFrom(a).getTimeseriesList();
        for (Types.TimeSeries timeSeries : timeSeriesList) {
            List<Types.Sample> samples = timeSeries.getSamplesList();
            List<Types.Label> labels = timeSeries.getLabelsList();
            Map<String, Object> map = new LinkedHashMap<>();
            for (Types.Label label : labels) {
                    map.put(label.getName(), label.getValue());
            }
            for (Types.Sample sample : samples) {
                map.put("timestamp", sample.getTimestamp());
                map.put("value", sample.getValue());
            }
            System.out.println(map);
        }
    }

}

//控制台打印：
labels {
  name: "__name__"
  value: "mysql_global_variables_query_cache_min_res_unit"
}
labels {
  name: "instance"
  value: "192.168.31.40:9101"
}
labels {
  name: "job"
  value: "ser4"
}
samples {
  value: 4096.0
  timestamp: 1601196751516
}

---

英文文章参考：https://atchison.dev/prometheus-remote-write-in-java/

Prometheus is a time series database. I am currently using Prometheus with Kafka, having the jmx agent expose beans for Prometheus to ingest with Grafana for visualization. It’s great, but temporary. Prometheus is designed to be an ephemeral cache and does not try to solve distributed data storage. To that end, Prometheus provides a “remote_write” configuration option to POST data sampling to an endpoint for the ingest point for persistence.

The protocol is a snappy compressed protobuf that contains the data sampling.

Most of the ecosystem around this feature is Go based. To the point where Googling around on how to do this in other languages suggests stripping out gogolang items from the protobuf spec files. I could not find a single example of doing this in another language, so here is an example in Java for posterity.

Get the .proto files Prometheus uses for the “remote_write” protobuf. They can be found here. Make sure you get the ones that match the version of the Prometheus server you are running.

remote.proto
types.proto

You will then need gogo.proto, which can be found here.

In summary, you now have remote, types, and gogo .proto files.

Compile these .proto files with protoc into your language of choice. For this tutorial I am going to use Java. You could build protoc from source, but I found it easier to just download the precompiled binary. They can be found here. You want the protoc-3.11.2-PLATFORM.zip

• Directory layout for these instructions:

./
protoc/ //unzipped binaries from above
output/ //destination for our language files
imports/
    remote.proto
    types.proto
    gogoproto/
        gogo.proto

• Generate Java code:

./protoc/bin/protoc --proto_path=./imports --java_out=./java_output/ imports/types.proto
./protoc/bin/protoc --proto_path=./imports --java_out=./java_output/ imports/remote.proto
./protoc/bin/protoc --proto_path=./imports --java_out=./java_output/ imports/gogoproto/gogo.proto

Order doesn’t matter, for some reason it won’t import generated items…? I assume this is because I don’t know enough about the protobuf system.

• In any event, you should then see the following:

ls -R java_output/
java_output/:
  com prometheus
java_output/com:
  google
java_output/com/google:
  protobuf
java_output/com/google/protobuf:
  GoGoProtos.java
java_output/prometheus:
  Remote.java Types.java

Now let’s write a server to receive requests from Prometheus. Jetty is nice for a quick prototype.

You’ll need the following additional dependencies for your project:

<!-- protobuf -->
<dependency>
   <groupId>com.google.protobuf</groupId>
   <artifactId>protobuf-java</artifactId>
   <version>3.11.1</version>
</dependency>
<dependency>
   <groupId>com.google.protobuf</groupId>
   <artifactId>protobuf-java-util</artifactId>
   <version>3.11.1</version>
</dependency>

<!-- snappy compression -->
<dependency>
   <groupId>org.xerial.snappy</groupId>
   <artifactId>snappy-java</artifactId>
   <version>1.1.7.3</version>
</dependency>

Handler for our prometheus metric protobuf data:

public class PrometheusHandler extends AbstractHandler {

    private static final Logger logger = LoggerFactory.getLogger(PrometheusHandler.class);

    private static JsonFormat.Printer JSON_PRINTER = JsonFormat.printer();

    public PrometheusHandler() {
        super();
    }

    @Override
    public void handle(String target, Request baseRequest,
                       HttpServletRequest request, HttpServletResponse response) throws IOException {

        try (InputStream is = request.getInputStream()) {
            ByteArrayOutputStream buffer = new ByteArrayOutputStream();
            int nRead;
            byte[] data = new byte[1024];
            while ((nRead = is.read(data, 0, data.length)) != -1) {
                buffer.write(data, 0, nRead);
            }

            buffer.flush();
            Remote.WriteRequest writeRequest = Remote.WriteRequest.parseFrom(Snappy.uncompress(buffer.toByteArray()));
            String json = JSON_PRINTER.print(writeRequest);
            logger.info(json);
        }
        catch (IOException e) {
            throw e;
        }
    }
}

Main Server:

public class MetricsReporter {

    private static final Logger logger = LoggerFactory.getLogger(MetricsReporter.class);

    private static Server createServer(final int port){
        Server server = new Server(port);
        server.setHandler(new PrometheusHandler());
        return server;
    }

    public static void main(String[] args) throws Exception {
        logger.info("Starting metrics reporting server: ");
        Server server = createServer(8000);
        server.start();
        server.join();
    }
}

Set the remote_write config section in prometheus.yml for your Prometheus server:

remote_write:
  - url: 'http://localhost:8000/receive'

Start the Jetty server and start/restart Prometheus. You should shortly see data coming in fairly frequently.

The original documentation and suggestions indicated that this was not doable outside of Go, which seemed weird considering the whole point of an agnostic data format like protobuf is specifically to avoid eco system lockdown.

Note: This post is an amalgamtion of two previous posts on the subject, outling the methodology using python3 and then an implementation in Java.