pyspark 加载jar,创建上下文后，将jar文件添加到pyspark

最新推荐文章于 2021-02-16 05:58:36 发布

Zero601

最新推荐文章于 2021-02-16 05:58:36 发布

阅读量158

点赞数

文章标签： pyspark 加载jar

在不负责创建SparkSession的环境中，通过Python接口调用Spark Scala上下文的addJAR方法来加载包含自定义函数的jar包。解决方案是利用sparksession._jsc.addJAR完成任务。

摘要由CSDN通过智能技术生成

I am using pyspark from a notebook and I do not handle the creation of the SparkSession.

I need to load a jar containing some functions I would like to use while processing my rdds. This is something which you can easily do using --jars which I cannot do in my particular case.

Is there a way to access the spark scala context and call the addJar method? I tried to use the JavaGateway (sparksession._jvm...) but have not been successful so far. Any idea?

Thanks

Guillaume

解决方案

sparksession._jsc.addJar does the job.