spark
kieron_wei
这个作者很懒,什么都没留下…
展开
-
spark 常用配置参数调优
spark 参数调优 spark问题:(spark.sql.hive.metastore.version,1.2.1) ERROR YarnScheduler: Lost executor 53 on node100p32: Container killed by YARN for exceeding memory limits. 10.0 GB of 10 GB physical memory used. Consider boosting spark.yarn.executor.memoryOv原创 2020-10-03 01:18:24 · 1559 阅读 · 0 评论 -
好用到不行的 SparkSQL 开窗函数
好用到不行的 SQL开窗函数 To use window functions, users need to mark that a function is used as a window function by either Adding an OVER clause after a supported function in SQL, e.g. avg(revenue) OVER (…);...原创 2019-11-15 14:11:37 · 437 阅读 · 0 评论 -
spark rdd 基础概要1
1 RDD基础 1.1 transformation 2.2 action 1.1 transformation 1.1.1 基础transformation map flatMap filter disinct mapPartition mapPartitionsWithIndex union intersection substract coalesce repartition r...原创 2019-11-14 22:20:11 · 244 阅读 · 0 评论