pyspark 数据写入hive_Pyspark将数据写入hi

最新推荐文章于 2024-08-10 11:07:33 发布

LeoShaoQiang

最新推荐文章于 2024-08-10 11:07:33 发布

阅读量533

点赞数

文章标签： pyspark 数据写入hive

本文链接：https://blog.csdn.net/weixin_42502382/article/details/111919416

版权

下面是我将数据写入配置单元的代码from pyspark import since,SparkContext as sc

from pyspark.sql import SparkSession

from pyspark.sql.functions import _functions , isnan

from pyspark.sql import SQLContext

from pyspark.sql.types import *

from pyspark import HiveContext as hc

spark = SparkSession.builder.appName("example-spark").config("spark.sql.crossJoin.enabled","true").config('spark.sql.warehouse.dir',"file:///C:/spark-2.0.0-bin-hadoop2.7/bin/metastore_db/spark-warehouse").config('spark.rpc.message.maxSize','1536').getOrCreate()

Name= spark.read.csv("file:///D:/valid.csv", header="true",inferSchema =

True,sep=',')

join_df=join_df.where("LastName != ''").show()

join_df.registerTempTable("test")

hc.sql("CREATE TABLE dev_party_tgt_repl STORED AS PARQUETFILE AS SELECT * from dev_party_tgt")

在执行上述代码后，我得到下面的错误

^{pr2}$

我的系统环境详细信息：操作系统：Windows

日蚀霓虹灯

Spark版本：2.0.0