问题一
如下报错,
第4行定义的action_time 的形式如下:
StructField("action_time", StringType, nullable = false)
即不允许为空,但是转换的数据中有对应不上的。所以会报错。
Caused by: java.lang.RuntimeException: The 4th field 'action_time' of input row cannot be null.
这里的4th 其实指的是第五列的内容。
问题二
如下报错:
Caused by: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of string
spark 的schema 定义和对应的数据对应不上,所以报错。
val schema = Array(
StructField("session_id",StringType, nullable = false),
StructField("user_id", LongType, nullable = false),
StructField("page_id", LongType, nullable = false),
StructField("action_time", StringType, nullable = false),
StructField("search_keyword", StringType, nullable = true),
StructField("click_category_id", LongType, nullable = true),
StructField("click_procduct_id", LongType, nullable = true),
StructField("order_category_ids", StringType, nullable = true),
StructField("order_product_ids", StringType, nullable = true),
StructField("pay_category_ids", StringType, nullable = true),
StructField("pay_product_ids", StringType, nullable = true),
StructField("date", StringType, nullable = false)
)