scala中的foldLeft学习

6 篇文章 0 订阅
1 篇文章 0 订阅

闲来无事,在stackoverflow上看代码玩,偶尔发现一个之前没有用过的函数--foldLeft函数。现做记录如下:

val sourceDF = Seq(
  ("  p a   b l o", "Paraguay"),
  ("Neymar", "B r    asil")
).toDF("name", "country")

val actualDF = Seq(
  "name",
  "country"
).foldLeft(sourceDF) { (memoDF, colName) =>
  memoDF.withColumn(
    colName,
    regexp_replace(col(colName), "\\s+", "")
  )
}

actualDF.show()
+------+--------+
|  name| country|
+------+--------+
| pablo|Paraguay|
|Neymar|  Brasil|
+------+--------+
val sourceDF = Seq(
  ("funny", "joke")
).toDF("A b C", "de F")

sourceDF.show()
+-----+----+
|A b C|de F|
+-----+----+
|funny|joke|
+-----+----+
val actualDF = sourceDF
  .columns
  .foldLeft(sourceDF) { (memoDF, colName) =>
    memoDF
      .withColumnRenamed(
        colName,
        colName.toLowerCase().replace(" ", "_")
      )
  }

actualDF.show()
+-----+----+
|a_b_c|de_f|
+-----+----+
|funny|joke|
+-----+----+
import org.apache.spark.sql.DataFrame

def snakeCaseColumns(df: DataFrame): DataFrame = {
  df.columns.foldLeft(df) { (memoDF, colName) =>
    memoDF.withColumnRenamed(colName, toSnakeCase(colName))
  }
}

def toSnakeCase(str: String): String = {
  str.toLowerCase().replace(" ", "_")
}
val sourceDF = Seq(
  ("funny", "joke")
).toDF("A b C", "de F")
val actualDF = sourceDF.transform(snakeCaseColumns)
actualDF.show()
+-----+----+
|a_b_c|de_f|
+-----+----+
|funny|joke|
+-----+----+

参考博客为

1、How can I concat several float columns into one ArrayType(FloatType()) in spark DataFrame?

2、Performing operations on multiple columns in a Spark DataFrame with foldLeft

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值