spark直接写hive表速度慢

1. sparksql动态分区直接写入hive表速度慢

1 动态写入速度慢的sql

set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table ssjt_test partition(dt) select a,dt from ssjt.test2 where dt>='20200801';

2. 文件方式写入后, alter方式添加分区
1) 过滤出要删除的目录;
2) 删除要重新覆盖的目录;
3) 文件方式写入(append)
import org.apache.spark.sql._
val out_path = "s3://transsion-bigdata-warehouse/test/shy/bak4"
df.write.mode( "append" ).option( "compression", "snappy" ).partitionBy( "dt" ).parquet( out_path )
4) 添加分区, 如果以前添加过可以省略;

2 新的解决思路参考

spark写入hive表发生NativeFileSystem:rename过程_盛源的博客-CSDN博客

---

-

-

-