报错信息如下:
Exception in thread "main" org.apache.flink.api.common.typeutils.CompositeType$InvalidFieldReferenceException: Cannot reference field by field expression on GenericType<cn.fancychuan.SensorReading>Field expressions are only supported on POJO types, tuples, and case classes. (See the Flink documentation on what is considered a POJO.)
at org.apache.flink.streaming.util.typeutils.FieldAccessorFactory.getAccessor(FieldAccessorFactory.java:193)
at org.apache.flink.streaming.api.functions.aggregation.ComparableAggregator.<init>(ComparableAggregator.java:67)
at org.apache.flink.streaming.api.datastream.KeyedStream.max(KeyedStream.java:836)
at cn.fancychuan.JavaTransformApp.main(JavaTransformApp.java:65)
报错的代码如下:
package cn.fancychuan;
import cn.fancychuan.SensorReading;
import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;
public class JavaTransformApp {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
DataStream<String> inputStream = env.readTextFile("E:\\JavaWorkshop\\bigdata-learn\\flink\\src\\main\\resources\\sensor.txt");
DataStream<SensorReading> dataStream = inputStream.map(line -> {
String[] items = line.split(",");
return new SensorReading(items[0], Long.parseLong(items[1]), new Double(items[2]));
});
KeyedStream<SensorReading, Tuple> keyedStream = dataStream.keyBy("id");
// 另一种写法: KeyedStream<SensorReading, String> keyedStream1 = dataStream.keyBy(data -> data.getId());
KeyedStream<SensorReading, String> keyedStream1 = dataStream.keyBy(SensorReading::getId);
SingleOutputStreamOperator<SensorReading> maxTemperature = keyedStream.max("temperature");
SingleOutputStreamOperator<SensorReading> maxByTemperature = keyedStream.maxBy("temperature");
maxTemperature.print("keyBy-max");
env.execute();
}
}
其中SensorReading类如下:
package cn.fancychuan;
public class SensorReading {
private String id;
private Long timestamp;
private Double temperature;
public SensorReading(String id, Long timestamp, Double temperature) {
this.id = id;
this.timestamp = timestamp;
this.temperature = temperature;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public Long getTimestamp() {
return timestamp;
}
public void setTimestamp(Long timestamp) {
this.timestamp = timestamp;
}
public Double getTemperature() {
return temperature;
}
public void setTemperature(Double temperature) {
this.temperature = temperature;
}
@Override
public String toString() {
return "SensorReading{" +
"id='" + id + '\'' +
", timestamp=" + timestamp +
", temperature=" + temperature +
'}';
}
}
这个类报错的原因是因为SensorReading类加了构造器
public SensorReading(String id, Long timestamp, Double temperature)
但是缺少无参构造器,在java中,如果定义了有参构造器,那么就不再会自动加上默认的无参构造器了。而在keyBy中,flink支持的需要是POJO类,或者tuple、样例类。在这里需要是POJO类,因此加上无参构造器即可。如下
package cn.fancychuan;
public class SensorReading {
private String id;
private Long timestamp;
private Double temperature;
/**
* 注意要加上这个默认无参构造器
*/
public SensorReading() {
}
public SensorReading(String id, Long timestamp, Double temperature) {
this.id = id;
this.timestamp = timestamp;
this.temperature = temperature;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public Long getTimestamp() {
return timestamp;
}
public void setTimestamp(Long timestamp) {
this.timestamp = timestamp;
}
public Double getTemperature() {
return temperature;
}
public void setTemperature(Double temperature) {
this.temperature = temperature;
}
@Override
public String toString() {
return "SensorReading{" +
"id='" + id + '\'' +
", timestamp=" + timestamp +
", temperature=" + temperature +
'}';
}
}
标准的POJO类:
1. 所有成员变量都是私有的,用private修饰
2. 每个成员变量都有对应的getter和setter
3. 有一个全参数的构造方法
4. 有一个无参的构造方法