pyspark 加载jar_将Jar添加到独立的pyspark

最新推荐文章于 2024-07-23 10:32:05 发布

宝源冷气工程

最新推荐文章于 2024-07-23 10:32:05 发布

阅读量788

点赞数

文章标签： pyspark 加载jar

本文链接：https://blog.csdn.net/weixin_35029593/article/details/113009757

版权

在不使用spark-submit启动的PySpark程序中，如何添加如Databricks csv jar这样的依赖？可以通过设置`spark.jars.packages`配置项来实现，例如：`config('spark.jars.packages', 'org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0')`。这种方法适用于Jupyter Notebook环境，无需修改环境变量或配置文件。" 128730735,8348142,C++实现自旋锁：原子操作与并发控制,"['c++', '并发编程', '自旋锁', '锁机制', '原子操作']

摘要由CSDN通过智能技术生成

I'm launching a pyspark program:

$ export SPARK_HOME=

$ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip

$ python

And the py code:

from pyspark import SparkContext, SparkConf

SparkConf().setAppName("Example").setMaster("local[2]")

sc = SparkContext(conf=conf)

How do I add jar dependencies such as the Databricks csv jar? Using the command line, I can add the package like this:

$ pyspark/spark-submit --packages com.databricks:spark-csv_2.10:1.3.0

But I'm not using any of these. The program is part of a larger workflow that is not using spark-submit I should be able to run my ./foo.py program and it should just work.

I know you can set