spark 从Rdd 构造df 报错_long is not a valid external type for schema of st-CSDN博客

本文链接：https://blog.csdn.net/eases_stone/article/details/80506038

问题一

如下报错，
第4行定义的action_time 的形式如下：
 StructField("action_time", StringType, nullable = false)
 即不允许为空，但是转换的数据中有对应不上的。所以会报错。
Caused by: java.lang.RuntimeException: The 4th field 'action_time' of input row cannot be null.
这里的4th 其实指的是第五列的内容。

问题二

如下报错：
Caused by: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of string

spark 的schema 定义和对应的数据对应不上，所以报错。
val schema = Array(
            StructField("session_id",StringType, nullable = false),
            StructField("user_id", LongType, nullable = false),
            StructField("page_id", LongType, nullable = false),
            StructField("action_time", StringType, nullable = false),
            StructField("search_keyword", StringType, nullable = true),
            StructField("click_category_id", LongType, nullable = true),
            StructField("click_procduct_id", LongType, nullable = true),
            StructField("order_category_ids", StringType, nullable = true),
            StructField("order_product_ids", StringType, nullable = true),
            StructField("pay_category_ids", StringType, nullable = true),
            StructField("pay_product_ids", StringType, nullable = true),
            StructField("date", StringType, nullable = false)
        )