Spark保存HDFS示例

最新推荐文章于 2022-05-22 22:21:18 发布

weixin_34082695

最新推荐文章于 2022-05-22 22:21:18 发布

阅读量969

点赞数

文章标签：大数据 python java

原文链接：https://my.oschina.net/u/778683/blog/2982869

版权

2019独角兽企业重金招聘Python工程师标准>>>

def saveAsNewAPIHadoopFile(
path: String,
keyClass: Class[_],
valueClass: Class[_],
outputFormatClass: Class[_ <: NewOutputFormat[_, _]],
conf: Configuration = self.context.hadoopConfiguration): Unit = self.withScope {
// Rename this as hadoopConf internally to avoid shadowing (see SPARK-2038).
val hadoopConf = conf
val job = NewAPIHadoopJob.getInstance(hadoopConf)
job.setOutputKeyClass(keyClass)
job.setOutputValueClass(valueClass)
job.setOutputFormatClass(outputFormatClass)
val jobConfiguration = job.getConfiguration
jobConfiguration.set("mapreduce.output.fileoutputformat.outputdir", path)
saveAsNewAPIHadoopDataset(jobConfiguration)
}