Spark问题12之kryoserializer shuffle size 不够,出现overflow

最新推荐文章于 2024-05-07 14:02:37 发布
最新推荐文章于 2024-05-07 14:02:37 发布
阅读量547 收藏
点赞数
分类专栏： spark
原文链接：https://blog.csdn.net/xubo245/article/details/60572078
版权

 

更多代码请见：https://github.com/xubo245/SparkLearning

Spark生态之Alluxio学习 版本：alluxio（tachyon） 0.7.1，spark-1.5.2,hadoop-2.6.0

1.问题描述

1.1

运行cs-bwamem是出现序列化shuffle overflow问题，主要是需要输出sam到本地，文件比较大，默认的是：

spark.kryoserializer.buffer.max 64m

而实际大于2G，所以不够。spark webUI显示14.5 GBinput。。。。 而且是collect操作

2.脚本运行

hadoop@Master:~/disk2/xubo/project/alignment/cs-bwamem$ cat csbwamemAlignP1Test10sam.sh 
#for t in 1 2 3 4 5 6 7 8 9 10 15 20 30 40 50 60 70 80 90 100 120 140 160 180 200 300 400 500 600 700 800 900 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
#for t in 1 10 50 100 400 1000 5000 10000
#for t in {90..400..10}
for t in 100
do
for k in {7..9}
do 
#for j in 10000 100000 1000000 10000000
for j in 10000000
do
for i in 50
do
echo $i
echo $j
echo 't'$t
echo 'k'$k
#fq='g38L'$i'c'$j'Nhs20Paired'$k'.fq'
#fq0='g38L'$i'c'$j'Nhs20Paired*.fastq'
#fq1='/xubo/alignment/sparkBWA/g38L'$i'c'$j'Nhs20Paired1.fastq'
#fq2='/xubo/alignment/sparkBWA/g38L'$i'c'$j'Nhs20Paired2.fastq'
#out='g38L'$i'c'$j'Nhs20Paired12.sam'
#out='/xubo/project/alignment/cs-bwamem/input/fastq/newg38L'$i'c'$j'Nhs20Paired12P64bn200000000t'$t'k'$k'sbatch.adam'
out='/home/hadoop/disk2/xubo/project/alignment/cs-bwamem/newg38L'$i'c'$j'Nhs20Paired12P64bn200000000.sam'
file='/xubo/project/alignment/cs-bwamem/input/fastq/newg38L'$i'c'$j'Nhs20Paired12P64bn200000000.fastq'
#out='/xubo/project/alignment/cs-bwamem/input/fastq/newg38L'$i'c'$j'Nhs20Paired12P1k'$k'.adam'
#file='/xubo/project/alignment/cs-bwamem/input/fastq/newg38L'$i'c'$j'Nhs20Paired12P1.fastq'

echo $file
echo $out

spark-submit --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 20 --executor-cores 2 --executor-memory 20G \
--master spark://219.219.220.149:7077 /home/hadoop/disk2/xubo/tools/cloud-scale-bwamem-0.2.2/target/cloud-scale-bwamem-0.2.2-assembly.jar \
cs-bwamem -bfn 1 -bPSW 1 -sbatch $t -bPSWJNI 1  -oChoice 1 -oPath $out -localRef 1 \
-jniPath /home/hadoop/disk2/xubo/tools/cloud-scale-bwamem-0.2.2/target/jniNative.so \
-isSWExtBatched 1  1 \
/home/hadoop/disk2/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta  $file

#spark-submit --executor-memory 6g --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 20 --master spark://219.219.220.149:7077  --conf spark.driver.host=219.219.220.149 --conf spark.driver.cores=4 --conf spark.driver.maxResultSize=6g --conf spark.storage.memoryFraction=0.7  --conf spark.akka.threads=2 --conf spark.akka.frameSize=1024 /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar merge hdfs://219.219.220.149:9000 $file $out


#/xubo/alignment/sparkBWA/GRCH38chr1L3556522N10L50paired1.fastq /xubo/alignment/sparkBWA/GRCH38chr1L3556522N10L50paired2.fastq \
#/xubo/alignment/output/sparkBWA/datatestLocalGRCH38chr1L3556522N10L50paired12YarnMaster

done 
done
done
done
#--master spark://219.219.220.149:7077 /home/hadoop/disk2/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar \
#--master spark://219.219.220.149:7077 /curr/pengwei/github/cloud-scale-bwamem/target/cloud-scale-bwamem-0.2.0-assembly.jar \

3.运行记录：

hadoop@Master:~/disk2/xubo/project/alignment/cs-bwamem$ ./csbwamemAlignP1Test10sam.sh > csbwamemAlignP1Test10samtime201702281645.txt
[Stage 2:>                                                        (0 + 16) / 64]17/02/28 16:57:37 ERROR TaskSetManager: Task 9 in stage 2.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 2.0 failed 4 times, most recent failure: Lost task 9.3 in stage 2.0 (TID 180, Mcnode4): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 109. To avoid this, increase spark.kryoserializer.buffer.max value.
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:263)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:909)
		at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
		at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
		at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
		at org.apache.spark.rdd.RDD.collect(RDD.scala:908)
		at cs.ucla.edu.bwaspark.FastMap$.memPairEndMapping(FastMap.scala:397)
		at cs.ucla.edu.bwaspark.FastMap$.memMain(FastMap.scala:144)
		at cs.ucla.edu.bwaspark.BWAMEMSpark$.main(BWAMEMSpark.scala:318)
		at cs.ucla.edu.bwaspark.BWAMEMSpark.main(BWAMEMSpark.scala)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:606)
		at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 109. To avoid this, increase spark.kryoserializer.buffer.max value.
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:263)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

参考

【1】http://spark.apache.org/docs/1.5.2/programming-guide.html
【2】https://github.com/xubo245/SparkLearning
zxfBdd
关注
专栏目录
Spark: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow
杨鑫newlife的专栏
04-23 1396
运行Spark程序的时候报出：Exception in thread "main" java.sql.SQLException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3107 in stage 308.0 failed 4 times, most recent failure: Lost t...
spark解决org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow
onway_goahead的博客
08-07 882
pyspark --queue default\ --driver-memory 10G \ --executor-cores 6 \ --executor-memory 10G \ --executor-cores 6 \ --conf spark.kryoserializer.buffer.max=256m \ --conf spark.kryoserializer.buffer=64m \ --conf spark.driver.maxResultSize=4096m \ --conf spark.e
【spark实战：报错】spark序列化报错：Kryo serialization failed: Buffer overflow
最新发布
hiliang521的博客
05-07 489
【报错】spark序列化报错：Kryo serialization failed: Buffer overflow
spark解决 org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow
热门推荐
xinganshenguang的博客
01-17 1万+
使用spark sql 的 thrift jdbc接口查询数据时报这个错误Exception in thread "main" java.sql.SQLException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3107 in stage 308.0 failed 4 times, most r
Spark 的两种核心 Shuffle 详解.pdf
10-08
Spark Shuffle 分为两种：一种是基于 Hash 的 Shuffle；另一种是基于 Sort 的 Shuffle。
Spark 优化之：Shuffle优化
AriesLY0411的博客
04-26 926
Shuffle调优 1. map和reduce端缓冲区大小 在Spark任务运行过程中，如果shuffle的map端处理的数据量比较大，但是map端缓冲的大小是固定的，可能会出现map端缓冲数据频繁spill溢写到磁盘文件中的情况，使得性能非常低下，通过调节map端缓冲的大小，可以避免频繁的磁盘IO操作，进而提升Spark任务的整体性能。 map端缓冲的默认配置是32KB，如果每个task处理640KB的数据，那么会发生640/32 = 20次溢写，如果每个task处理64000KB的数据，即会发生6
Spark的Shuffle总结分析
01-07
一、shuffle原理分析 1.1 shuffle概述 Shuffle就是对数据进行重组，由于分布式计算的特性和要求，在实现细节上更加繁琐和复杂。 在MapReduce框架，Shuffle是连接Map和Reduce之间的桥梁，Map阶段通过shuffle读取数据...
Spark源码系列（六）Shuffle的过程解析
03-03
Spark大会上，所有的演讲嘉宾都认为shuffle是最影响性能的地方，但是又无可奈何。之前去百度面试hadoop的时候，也被问到了这个问题，直接回答了不知道。这篇文章主要是沿着下面几个问题来开展：shuffle过程的划分？...
spark shuffle原理
04-21
理解Spark Shuffle的原理对于优化大数据应用的性能至关重要，开发者可以通过调整Shuffle相关的配置参数，如buffer size、shuffle partition数量等，来平衡内存使用、磁盘I/O和网络传输，以达到最佳的处理效果。
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow
康师傅没有眼泪
10-09 1840
运行Spark任务报错如下： 21/10/09 14:56:32 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 4) org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 93. To avoid this, increase spark.kryoserializer.buffer....
Spark报错处理系列之：Kryo serialization failed: Buffer overflow. Available: 0, required: 12678.
zhengzaifeidelushang的博客
12-16 539
Spark报错处理系列之：Kryo serialization failed: Buffer overflow. Available: 0, required: 12678. To avoid this, increase spark.kryoserializer.buffer.max value.
错误：Caused by:org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow.Available: ...
angou6476的博客
01-26 281
这个是写入Redis时用的序列化器，然后错误提示是超过了大小限制，把配置调大即可。 .set("spark.kryoserializer.buffer.max","128"); 如果没有配置，那么找一下看下有没有硬编码写了大小的范围导致的。 参考： http://blog.csdn.net/keyuquan/article/details/73379955 htt...
org.apache.spark.SparkException: Kryo serialization failed
亮仔的专栏
02-22 873
在sparksql中显示的指定了mapjoin，导致广播的数据量太大，导致序列化超过指定大小。去除显示指定mapjoin sql如下： with einfo as     (        select            E6.EMP_NO    ,            E6.TEAM_ID       ,            E6.TEAM_NAME         ...
在使用spark时，碰到内存不足问题
qq_36055407的博客
01-14 4880
在使用spark时，默认内存的分配为1GB，但是由于学习环境有限，服务器内存太小不能满足最小内存（1GB）。当在启动spark-shell后，可能过会儿就被kill掉了，比如: scala> bin/spark-shell: line 44: 9896 Killed "${SPARK_HOME}"/bin/spark-submit --class org...
spark debug org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow.
LXWalaz1s1s的博客
05-29 768
背景   今天在spark-shell临时性导出300万数数据生成csv文件的时候，其中我启动spark-shell的语句如下； # 用yarn模式运行spark，client模式提交任务，运行在etl队列，分配4个executors 每个1核 2G内存 spark-shell --master yarn --deploy-mode client --executor-memory 2G --executor-cores 1 --num-executors 4 --queue etl   生成临时性csv
spark Kryo serialization failed: Buffer overflow 错误
weixin_33940102的博客
12-14 2009
今天在写spark任务的时候遇到这么一个错误，我的spark版本是1.5.1. 1 Exception in thread "main" com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 124 2 at com.esotericsoftware.kryo.io.Out...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值