scala语言下使用protobuf协议解析数据

scala语言下使用protobuf协议解析数据

背景

1. kafka数据源是protobuf格式，需要反序列化，并取出相关字段。
2. 已有xxx.proto文件，需要生成解析类。

存在的问题

1. protoc命令行，不支持直接生成scala文件
   
2. 网上寻找工具，转了一圈各大论坛，基本都是 先生成 java文件，然后再scala里面引用，这样不符合需求，不愿仅仅因为解析protobuf而引入java语言

解决

pom文件

 <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <scala.version>2.12.6</scala.version>
    <scala.compat.version>2.12</scala.compat.version>
    <spec2.version>4.2.0</spec2.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>com.thesamet.scalapb</groupId>
      <artifactId>scalapb-runtime_${scala.compat.version}</artifactId>
      <version>0.9.0-M5</version>
    </dependency>

  </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <extensions>
      <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>1.6.0</version>
      </extension>
    </extensions>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.3.2</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
            <configuration>
              <args>
                <arg>-dependencyfile</arg>
                <arg>${project.build.directory}/.scala_dependencies</arg>
              </args>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.21.0</version>
        <configuration>
          <!-- Tests will be run with scalatest-maven-plugin instead -->
          <skipTests>true</skipTests>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest-maven-plugin</artifactId>
        <version>2.0.0</version>
        <configuration>
          <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
          <junitxml>.</junitxml>
          <filereports>TestSuiteReport.txt</filereports>
          <!-- Comma separated list of JUnit test class names to execute -->
          <jUnitClasses>scalapb.demo.ProtoSuite</jUnitClasses>
        </configuration>
        <executions>
          <execution>
            <id>test</id>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <!-- Add protobuf-maven-plugin and provide ScalaPB as a code generation plugin -->
      <plugin>
          <groupId>com.github.os72</groupId>
          <artifactId>protoc-jar-maven-plugin</artifactId>
          <version>3.11.4</version>
          <executions>
              <execution>
                  <phase>generate-sources</phase>
                  <goals>
                      <goal>run</goal>
                  </goals>
              </execution>
          </executions>
          <configuration>
            <includeMavenTypes>transitive</includeMavenTypes>
            <outputTargets>
                <outputTarget>
                    <type>scalapb</type>
                    <outputOptions>grpc</outputOptions> <!-- more scalapb options can be added here -->
                    <pluginArtifact>com.thesamet.scalapb:protoc-gen-scala:0.9.6:sh:unix</pluginArtifact>
                </outputTarget>
            </outputTargets>
        </configuration>
      </plugin>
    </plugins>
  </build>

proto文件

引入proto文件，并修改包地址

生成源码

编译生成源码后，可以直接在所需代码里面引用