基因数据处理96之sparkBWA运行问题（yarn）

最新推荐文章于 2020-09-10 22:59:04 发布
最新推荐文章于 2020-09-10 22:59:04 发布
阅读量430 收藏
点赞数
分类专栏： 基因数据处理
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xubo245/article/details/78976864
版权 
    hadoop@Master:~/xubo/project/alignment/sparkBWA$ ./paired.sh 
    Using properties file: /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf
    Adding default property: spark.executor.extraJavaOptions=-Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
    Adding default property: spark.eventLog.enabled=true
    Adding default property: spark.eventLog.dir=file:///home/hadoop/Downloads/hangc/sparklog
    Adding default property: spark.eventLog.compress=true
    Adding default property: spark.yarn.executor.memoryOverhead=4704
    Parsed arguments:
      master                  yarn-client
      deployMode              null
      executorMemory          1500m
      executorCores           1
      totalExecutorCores      null
      propertiesFile          /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf
      driverMemory            1500m
      driverCores             null
      driverExtraClassPath    null
      driverExtraLibraryPath  null
      driverExtraJavaOptions  null
      supervise               false
      queue                   null
      numExecutors            32
      files                   null
      pyFiles                 null
      archives                file:/home/hadoop/xubo/project/alignment/sparkBWA/bwa.zip
      mainClass               SparkBWA
      primaryResource         file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar
      name                    SparkBWA
      childArgs               [-algorithm mem -reads paired -index /home/hadoop/xubo/project/alignment/sparkBWA/index/datatest.fasta /xubo/alignment/bwa/datatest.fq /xubo/alignment/bwa/datatest.fq /xubo/alignment/output/sparkBWA/datatest]
      jars                    null
      packages                null
      packagesExclusions      null
      repositories            null
      verbose                 true

    Spark properties used, including those specified through
     --conf and those from the properties file /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf:
      spark.driver.memory -> 1500m
      spark.eventLog.enabled -> true
      spark.eventLog.compress -> true
      spark.executor.extraJavaOptions -> -Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
      spark.yarn.executor.memoryOverhead -> 4704
      spark.eventLog.dir -> file:///home/hadoop/Downloads/hangc/sparklog


    Main class:
    SparkBWA
    Arguments:
    -algorithm
    mem
    -reads
    paired
    -index
    /home/hadoop/xubo/project/alignment/sparkBWA/index/datatest.fasta
    /xubo/alignment/bwa/datatest.fq
    /xubo/alignment/bwa/datatest.fq
    /xubo/alignment/output/sparkBWA/datatest
    System properties:
    spark.driver.memory -> 1500m
    spark.executor.memory -> 1500m
    spark.executor.instances -> 32
    spark.eventLog.enabled -> true
    spark.eventLog.compress -> true
    SPARK_SUBMIT -> true
    spark.executor.extraJavaOptions -> -Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
    spark.app.name -> SparkBWA
    spark.yarn.executor.memoryOverhead -> 4704
    spark.jars -> file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar
    spark.submit.deployMode -> client
    spark.yarn.dist.archives -> file:/home/hadoop/xubo/project/alignment/sparkBWA/bwa.zip
    spark.eventLog.dir -> file:///home/hadoop/Downloads/hangc/sparklog
    spark.master -> yarn-client
    spark.executor.cores -> 1
    Classpath elements:
    file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar


    16/06/19 21:44:45 ERROR YarnScheduler: Lost executor 2 on Mcnode3: remote Rpc client disassociated
    16/06/19 21:44:45 ERROR TaskSetManager: Task 0 in stage 2.0 failed 4 times; aborting job
    Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, Mcnode3): ExecutorLostFailure (executor 2 lost)
    Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
			at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
			at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
			at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
			at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
			at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
			at scala.Option.foreach(Option.scala:236)
			at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
			at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:909)
			at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
			at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
			at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
			at org.apache.spark.rdd.RDD.collect(RDD.scala:908)
			at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:338)
			at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:47)
			at BwaRDD.MapBwa(BwaRDD.java:108)
			at BwaInterpreter.RunBwa(BwaInterpreter.java:437)
			at SparkBWA.main(SparkBWA.java:30)
			at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
			at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
			at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.lang.reflect.Method.invoke(Method.java:606)
			at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    hadoop@Master:~/xubo/project/alignment/sparkBWA$ 


第二次运行：

hadoop@Master:~/xubo/project/alignment/sparkBWA$ ./paired.sh 
Using properties file: /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf
Adding default property: spark.executor.extraJavaOptions=-Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.eventLog.dir=file:///home/hadoop/Downloads/hangc/sparklog
Adding default property: spark.eventLog.compress=true
Adding default property: spark.yarn.executor.memoryOverhead=4704
Parsed arguments:
  master                  yarn-client
  deployMode              null
  executorMemory          1500m
  executorCores           1
  totalExecutorCores      null
  propertiesFile          /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf
  driverMemory            1500m
  driverCores             null
  driverExtraClassPath    null
  driverExtraLibraryPath  null
  driverExtraJavaOptions  null
  supervise               false
  queue                   null
  numExecutors            32
  files                   null
  pyFiles                 null
  archives                file:/home/hadoop/xubo/project/alignment/sparkBWA/bwa.zip
  mainClass               SparkBWA
  primaryResource         file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar
  name                    SparkBWA
  childArgs               [-algorithm mem -reads paired -index /home/hadoop/xubo/project/alignment/sparkBWA/index/datatest.fasta /xubo/alignment/bwa/datatest.fq /xubo/alignment/bwa/datatest.fq /xubo/alignment/output/sparkBWA/datatest]
  jars                    null
  packages                null
  packagesExclusions      null
  repositories            null
  verbose                 true

Spark properties used, including those specified through
 --conf and those from the properties file /home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.conf:
  spark.driver.memory -> 1500m
  spark.eventLog.enabled -> true
  spark.eventLog.compress -> true
  spark.executor.extraJavaOptions -> -Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
  spark.yarn.executor.memoryOverhead -> 4704
  spark.eventLog.dir -> file:///home/hadoop/Downloads/hangc/sparklog


Main class:
SparkBWA
Arguments:
-algorithm
mem
-reads
paired
-index
/home/hadoop/xubo/project/alignment/sparkBWA/index/datatest.fasta
/xubo/alignment/bwa/datatest.fq
/xubo/alignment/bwa/datatest.fq
/xubo/alignment/output/sparkBWA/datatest
System properties:
spark.driver.memory -> 1500m
spark.executor.memory -> 1500m
spark.executor.instances -> 32
spark.eventLog.enabled -> true
spark.eventLog.compress -> true
SPARK_SUBMIT -> true
spark.executor.extraJavaOptions -> -Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build/bwa.zip
spark.app.name -> SparkBWA
spark.yarn.executor.memoryOverhead -> 4704
spark.jars -> file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar
spark.submit.deployMode -> client
spark.yarn.dist.archives -> file:/home/hadoop/xubo/project/alignment/sparkBWA/bwa.zip
spark.eventLog.dir -> file:///home/hadoop/Downloads/hangc/sparklog
spark.master -> yarn-client
spark.executor.cores -> 1
Classpath elements:
file:/home/hadoop/xubo/project/alignment/sparkBWA/SparkBWA.jar


16/06/19 21:57:49 ERROR YarnScheduler: Lost executor 1 on Mcnode4: remote Rpc client disassociated
16/06/19 21:57:55 ERROR YarnScheduler: Lost executor 4 on Mcnode2: remote Rpc client disassociated
16/06/19 21:58:11 ERROR YarnScheduler: Lost executor 7 on Mcnode4: remote Rpc client disassociated
16/06/19 21:58:11 ERROR TaskSetManager: Task 0 in stage 2.1 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.1 failed 4 times, most recent failure: Lost task 0.3 in stage 2.1 (TID 11, Mcnode3): java.lang.UnsatisfiedLinkError: no bwa in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
    at java.lang.Runtime.loadLibrary0(Runtime.java:849)
    at java.lang.System.loadLibrary(System.java:1088)
    at BwaJni.<clinit>(BwaJni.java:44)
    at Bwa.run(Bwa.java:443)
    at BwaRDD$BwaAlignment.call(BwaRDD.java:283)
    at BwaRDD$BwaAlignment.call(BwaRDD.java:173)
    at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction2$1.apply(JavaPairRDD.scala:1024)
		at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
		at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$18.apply(RDD.scala:727)
		at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$18.apply(RDD.scala:727)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:909)
		at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
		at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
		at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
		at org.apache.spark.rdd.RDD.collect(RDD.scala:908)
		at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:338)
		at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:47)
		at BwaRDD.MapBwa(BwaRDD.java:108)
		at BwaInterpreter.RunBwa(BwaInterpreter.java:437)
		at SparkBWA.main(SparkBWA.java:30)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:606)
		at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsatisfiedLinkError: no bwa in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
    at java.lang.Runtime.loadLibrary0(Runtime.java:849)
    at java.lang.System.loadLibrary(System.java:1088)
    at BwaJni.<clinit>(BwaJni.java:44)
    at Bwa.run(Bwa.java:443)
    at BwaRDD$BwaAlignment.call(BwaRDD.java:283)
    at BwaRDD$BwaAlignment.call(BwaRDD.java:173)
    at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction2$1.apply(JavaPairRDD.scala:1024)
		at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
		at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$18.apply(RDD.scala:727)
		at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$18.apply(RDD.scala:727)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
hadoop@Master:~/xubo/project/alignment/sparkBWA$

参考

【1】https://github.com/xubo245/AdamLearning
【2】https://github.com/bigdatagenomics/adam/ 
【3】https://github.com/xubo245/SparkLearning
【4】http://spark.apache.org
【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job  
【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver

研究成果：

【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)
【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).
【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).
【4】more: https://github.com/xubo245/Publications

Help

If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: xubo245@mail.ustc.edu.cn
Wechat: xu601450868
QQ: 601450868
KeepLearningBigData
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark算子中用到scalal类，由于未序列化报错
科研这件小事
11-23 6752
由于spark算子用到的class没有实现序列化，报错如下所示 15/11/23 14:43:47 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4) java.io.NotSerializableException: EntityMention Serialization stack: - object not ser
美团1万台 Hadoop 集群 YARN 的调优之路
01-07
背景 YARN作为Hadoop的资源管理系统，负责Hadoop集群上计算资源的管理和作业调度。 美团的YARN以社区2.7.1版本为基础构建分支。目前在YARN上支撑离线业务、实时业务以及机器学习业务。 离线业务主要运行的是Hive on MapReduce， Spark SQL为主的数据仓库作业。 实时业务主要运行Spark Streaming，Flink为主的实时流计算作业。 机器学习业务主要运行TensorFlow，MXNet，MLX（美团点评自研的大规模机器学习系统）等计算作业。 YARN面临高可用、扩展性、稳定性的问题很多。其中扩展性上遇到最严重的是集群和业务规模增长带来的调度器性
YARN框架原理及运行机制
01-27
YARN是Hadoop2.0中的资源管理系统，它的设计思想是将MRv1中的JobTracker拆分成两个独立的服务：一个全局的资源管理器ResourceManager和每个应用程序持有的ApplicationMaster。其中RM负责整个系统的资源管理和分配，AM负责单个应用程序的管理。YARN总体上仍然是Master/Slaver结构，在整个资源管理框架中，RM为Master，NM为Slaver，RM负责对各个NM上的资源进行统一管理和调度。当提交一个应用程序时，需要提供一个用以跟踪和管理这个程序的AM，它负责向RM申请资源，并要求NM启动占用一定资源的任务，不同的AM被分布在不同的节点上。
hadoop处理框架，hdfs，mapreduce，yarn
最新发布
08-29
hadoop处理框架，hdfs，mapreduce，yarn
Spark on Yarn模式的电信大数据处理平台.pdf
08-15
#资源达人分享计划#
基因数据处理98之SparkBWA运行时spark on Yarn问题日志完整记录
Keep Learning
01-11 963
脚本：hadoop@Mcnode1:~/xubo/tools/SparkBWA/build$ cat paired.sh spark-submit --class SparkBWA \ --master yarn-client \ --conf "spark.executor.extraJavaOptions=-XX:MaxPermSize=1024M" --driver-
Spark 2.x 性能调优-IBM
hery_csnd168的博客
03-13 1640
IBM Big Data PerformanceJesse Chen, [email protected] Troubleshooting Spark 2.x Building SparkRunning Spark-  ‘--verbose’-  Missing external JARs-  OOM on Spark driver-  OOM on executors-  GC pol...
SparkBWA编译失败报错解决
dltan
09-10 464
BWA、bwa基因软件在spark中运行，进行sparkbwa编译sparkbwa.jar包。解决编译报错问题。
搭建Spark所遇过的坑
asd315861547的博客
02-21 3168
一.经验1.Spark Streaming包含三种计算模式：nonstate .stateful .window2.kafka可通过配置文件使用自带的zookeeper集群3.Spark一切操作归根结底是对RDD的操作4.部署Spark任务，不用拷贝整个架包，只需拷贝被修改的文件，然后在目标服务器上编译打包。5.kafka的log.dirs不要设置成/tmp下的目录，貌似tmp目录有文件数和磁盘容量
基因数据处理102之SparkBWA本地运行100万条paired-reads实例
Keep Learning
01-11 1095
脚本：spark-submit --class SparkBWA \ --master local \ --archives bwa.zip \ SparkBWA.jar \ -algorithm mem -reads paired \ -index /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta \ -partitions
BWA编译（make）过程中出现的问题
weixin_44953795的博客
09-05 1191
会出现：致命错误、；zlib.h:没有那个文件或目录的问题。 查了一下百度用apt-get install zlib1g-dev 尝试解决这个问题，但是出现了bash：没有apt-get这个命令，因为我用的是centos7，这个系统要用yum来安装软件。 尝试用yum install zlib1g-dev，会显示没有可用安装包 zlib1g-dev这个问题。 又查了一下，用yum -y insta...
Spark基础，Local模式、Standalone模式、Yarn模式安装使用配置
Li_Jian_Hui_的博客
12-24 5368
第1章 Spark概述 1.1 什么是Spark 1、定义 Spark是-种基于内存的快速、通用、可扩展的大数据分析引擎。 2、历史 2009年诞生于加州大学伯克利分校AMPLab，项目采用Scala编写。 2010年开源， 2013年6月成为Apache孵化项目 2014年2月成为Apache顶级项目。 1.2 Spark内置模块 Spark Core：实现了Spark的基本功能，包含任务调...
基因数据处理117之重新多次运行SparkBWA Yarn集群
Keep Learning
06-03 580
更多代码请见：https://github.com/xubo245 基因数据处理系列 1.解释 重新运行，跟换了文件地址 后来终端，需要运行其他的 2.代码： hadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$ vi sparkBWA.sh for j in 10000 100000 1000000 1000000...
基因数据处理103之SparkBWAYarn模式运行100万条paired-reads实例
Keep Learning
01-11 552
脚本：spark-submit --class SparkBWA \ --master yarn-client \ --conf "spark.executor.extraJavaOptions=-Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build" \ --archives ./bwa.zip \ SparkBWA.jar \ -al
Spark On Yarn 运行模式（详细）
阿楠的博客
11-03 2482
Spark 在spark 中，支持4中运行模式： 1）Local:开发时使用 2）Standalone: 是spark 自带的，如果一个集群是standalong 的话，那么就需要在多台机器上同时部署spark 环境 3）Yarn:建议大家在生产上使用该模式，统一使用yarn 进行整个集群作业（mr、spark）的资源调度 4）Mesos ：跟yarn使用一样都是资源调度 不管使用什...
Spark 0.9.1和Shark 0.9.1分布式安装指南
weixin_34113237的博客
05-12 120
目录 目录 1 1.约定 1 2.安装Scala 1 2.1.下载 2 2.2.安装 2 2.3.设置环境变量 2 3.安装Spark 2 3.1.部署 2 3.2.下载 3 3.3.安装 3 3.4.配置 3 3.4.1.修改conf/spark-env.sh 3 3.4.2.修改conf/slaves 3 4.启动Spark 3 5...
hive中运行5000万数据加工，需要多少yarn资源
06-12
评估Hive任务所需的YARN资源需要考虑多个因素，包括数据量、数据处理方式、任务类型等等。给定数据量为5000万，可以根据以下指标来初步估计所需的YARN资源： 1. Mapper数量：Mapper数量通常可以根据数据量进行估算...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值