问题1:spark版本问题
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
一般碰到结尾为V,很可能是版本问题
解决办法:
<spark.version>2.4.0</spark.version>
附上代码地址:
user-profile-task0224: 用户画像scala计算代码
问题2:明明hue是可以直接创建的,但是spark代码一直报库不存在
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'user_profile0224' not found;
解决办法:
a.检查hadooponwindows-master环境变量
b.增加配置
val sparkConf: SparkConf = new SparkConf().setAppName("task_sql_app") .setMaster("local[*]") // .set("hive.metastore.uris","thrift://cdh01:9083") val sparkSession: SparkSession = SparkSession.builder() .config(sparkConf) .config("hive.metastore.uris","thrift://cdh01:9083") .config("spark.sql.warehouse.dir","hdfs://cdh01:8020/user/hive/warehouse") .enableHiveSupport() .getOrCreate()