Apache Beam 问题汇总

Apache Beam 问题汇总

  1. 泛型擦除问题
Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for ParDo(Anonymous)/ParMultiDo(Anonymous).output [PCollection@577127077]. Correct one of the following root causes:
  No Coder has been manually specified;  you may do so using .setCoder().
  Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder for java.lang.Object.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
  Using the default output Coder from the producing PTransform failed: PTransform.getOutputCoder called.
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:507)
	at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:286)
	at org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:117)
	at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifying(TransformHierarchy.java:226)
	at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:212)
	at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:469)
	at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:598)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
	at com.yss.beam.BeamMutiInput.main(BeamMutiInput.java:49)

这个问题一般发生在使用TupleTag的时候,需要在TupleTag初始化时加个{}即可

final TupleTag<Long> ATag = new TupleTag<Long>(){};
final TupleTag<Long> BTag = new TupleTag<Long>(){};

如果还不行,那就只能使用setCoder手动声明类型

PCollectionTuple resultTuple = pipeline
                .apply(Create.of("hello world", "hello flink", "hello flink spark"))
                .apply(ParDo.of(new DoFn<String, String>() {
                    @ProcessElement
                    public void processElement(@Element String word, MultiOutputReceiver out) {
                        if (word.endsWith("world")) {
                            out.get(endWithWorldTag).output(word);
                        } else if (word.endsWith("spark")) {
                            out.get(endWithSparkTag).output(word);
                        } else {
                            out.get(endWithFlinkTag).output(word);
                        }
                    }
                }).withOutputTags(endWithFlinkTag, TupleTagList.of(endWithSparkTag).and(endWithWorldTag)));
		// 在此处使用setCoder来声明类型
        resultTuple.get(endWithFlinkTag).setCoder(StringUtf8Coder.of()).apply(ParDo.of(new DoFn<String, String>() {
            @ProcessElement
            public void processElement(@Element String element){
                System.out.println("flink Tag " + element);
            }
        }));
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

EdwardsWang丶

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值