![](https://img-blog.csdnimg.cn/20201014180756916.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
大数据
merrily01
Talk is cheap 、show me the code .
展开
-
spark触发adaptive skewed join的例子code
1. 启动spark-shell,参数如下:spark-shell --conf spark.driver.allowMultipleContexts=true --conf spark.sql.adaptive.maxNumPostShufflePartitions=5 --conf spark.shuffle.statistics.verbose=true --conf spark.sql.adaptive.enabled=true --conf spark.sql.autoBroadcastJo原创 2020-11-18 23:34:29 · 397 阅读 · 0 评论 -
Spark Shuffle 例子
sc.parallelize(0 to 0xff, 100).map(i => (i ^ 2, i << 2, i & 2, i | 2)).repartition(100).count()原创 2020-11-09 14:41:09 · 208 阅读 · 0 评论 -
Spark On K8s Executor失败不删除(方便调试)
--conf spark.kubernetes.executor.deleteOnTermination=false原创 2020-08-10 17:09:27 · 513 阅读 · 0 评论 -
spark-sql显示表头header
spark-sql --hiveconf hive.cli.print.header=true原创 2020-05-25 18:40:52 · 4102 阅读 · 1 评论