spark java udf,Spark如何在Java中通过数据集调用UDF

What is exact translation of below scala code snippet in Java?

import org.apache.spark.sql.functions.udf

def upper(s:String) : String ={

s.toUpperCase

}

val toUpper = udf(upper _)

peopleDS.select(peopleDS(“name”),toUpper(peopledS(“name”))).show

please fill below missing statement java e.g.

import org.apache.spark.sql.api.java.UDF1;

UDF1 toUpper= new UDF1() {

public String call(final String str) throws Exception {

return str.toUpperCase(); }};

peopleDS.select(peopleDS.col("name"), /*how to run toUpper("name"))?????*/.show();

Note: Register udf then call using selectExpr works for me but I need something similar showed above.

Working example:

sqlContext.udf().register("toUpper",(String s)->s.toUpperCase(), DataTypes.StringType);

peopleDF.selectExpr("toUpper(name)","name").show();

解决方案

In java calling UDF without registration is not possible. Please check Using UDFs in Java without registration. Below is your UDF.

private static UDF1 toUpper = new UDF1() {

public String call(final String str) throws Exception {

return str.toUpperCase();

}

};

Register the UDF and you can use callUDF function.

import static org.apache.spark.sql.functions.callUDF;

import static org.apache.spark.sql.functions.col;

sqlContext.udf().register("toUpper", toUpper, DataTypes.StringType);

peopleDF.select(col("name"),callUDF("toUpper", col("name"))).show();

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值