java 事件冷却,apache flink中protobuf事件中的java问题反序列化事件

我在我的flink应用程序中 读取来自kinesis的事件。事件采用protobuf格式。如果我用'com.google.protobuf：原生动物-java:3.7.1“在flink应用程序中，我没有问题。但是如果我把它改成'com.google.protobuf：原生动物-java:3.10.0'我得到了上面的堆栈跟踪异常

java.lang.IncompatibleClassChangeError: class com.google.protobuf.Descriptors$OneofDescriptor has interface com.google.protobuf.Descriptors$GenericDescriptor as super class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)

at java.lang.Class.privateGetPublicMethods(Class.java:2902)

at java.lang.Class.privateGetPublicMethods(Class.java:2917)

at java.lang.Class.getMethods(Class.java:1615)

at org.apache.flink.api.java.typeutils.TypeExtractor.isValidPojoField(TypeExtractor.java:1786)

at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1856)

at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1746)

at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1643)

at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:921)

at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:781)

at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:735)

at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:731)

at org.apache.flink.api.common.typeinfo.TypeInformation.of(TypeInformation.java:211)

at org.apache.flink.api.java.typeutils.ListTypeInfo.(ListTypeInfo.java:45)

at com.bagi.streaming.serialization.ProtoSchema.getProducedType(ProtoSchema.java:40)

at org.apache.flink.streaming.connectors.kinesis.serialization.KinesisDeserializationSchemaWrapper.getProducedType(KinesisDeserializationSchemaWrapper.java:57)

at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.getProducedType(FlinkKinesisConsumer.java:363)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1456)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1414)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1396)

at com.bagi.streaming.StreamProcessor.getKinesisTrackingStream(StreamProcessor.java:101)

at com.bagi.streaming.StreamProcessor.getKinesisTrackingStream(StreamProcessor.java:110)

at com.bagi.streaming.StreamProcessor.consumeKinesis(StreamProcessor.java:117)

at com.bagi.streaming.StreamProcessor.main(StreamProcessor.java:80)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)

at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)

at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)

at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

我正在使用燧石@1.8.0和'com.twitter网站：冷却-方案：0.9.3'. 我正在我的mac上 local构建flink应用程序jar。我试过在3.10.0和3.7.1版本中使用protoc，以防发生问题。

这是我的反序列化程序

public class ProtoSchema implements DeserializationSchema> {

@Override

public List<Event> deserialize(byte[] message) throws IOException {

List<Event> events = new LinkedList<>();

InputStream inputStream = new ByteArrayInputStream(message);

while (true) {

Event event = Event.parseDelimitedFrom(inputStream);

if (event != null) {

events.add(event);

} else {

break;

}

}

return events;

}

@Override

public boolean isEndOfStream(List<Event> nextElement) {

return false;

}

@Override

public TypeInformation<List<Event>> getProducedType() {

return new ListTypeInfo<>(Event.class);

}

}

我要做的就是

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

Properties consumerConfig = new Properties();

consumerConfig.put(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");

consumerConfig.put(AWSConfigConstants.AWS_REGION, region);

consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "300");

consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_RETRIES, "10");

consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "5000");

consumerConfig.put(ConsumerConfigConstants.STREAM_INITIAL_POSITION, "LATEST");

env.addSource(new FlinkKinesisConsumer<>(name, new ProtoSchema(), consumerConfig)).name("KinesisSource");

env.getConfig().registerTypeWithKryoSerializer(Event.class, ProtobufSerializer.class);

事件.类使用从protobuf schema编译协议@3.10.0以及java@3.10.0协议