python 读取kafka 写hive_使用pyspark-stucked将数据从kafka写入hive

from pyspark.sql import SparkSession

from pyspark.sql.functions import explode

from pyspark.sql.functions import *

from pyspark.streaming.kafka import KafkaUtils

from os.path import abspath

warehouseLocation = abspath("spark-warehouse")

spark = SparkSession.builder.appName("sparkstreaming").getOrCreate()

df = spark.read.format("kafka").option("startingoffsets", "earliest").option("kafka.bootstrap.servers", "kafka-server1:66,kafka-server2:66").option("kafka.security.protocol", "SSL").option("kafka.ssl.keystore.location", "mykeystore.jks").option("kafka.ssl.keystore.password","mykeystorepassword").option("subscribe","json_stream").load().selectExpr("CAST(value AS STRING)")

json_schema = df.schema

df1 = df.select($"value").select(from_json,json_schema).alias("data").select("data.*")

上面的操作不起作用,但是在提取数据之后,我想将数据插入配置单元表。

提前感谢!:)