基因数据处理84之cs-bwamem处理小数据集

最新推荐文章于 2023-01-14 00:12:44 发布
最新推荐文章于 2023-01-14 00:12:44 发布
阅读量1k 收藏
点赞数
分类专栏： 基因数据处理 文章标签： 基因数据处理 spark adam
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xubo245/article/details/78898080
版权

1.使用的是之前的数据：
spark-submit –class cs.ucla.edu.bwaspark.BWAMEMSpark –master local[2] /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar upload-fastq 0 1 fastq/g38L100c50Nhs20.fastq /xubo/data/alignment/cs-bwamem/fastq/g38L100c50Nhs20upload2.fastq

2.之后有移动，所以比对时是：

hadoop@Master:~/xubo/project/alignment/CountAlignment$ spark-submit --executor-memory 2g --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 2 --master local[2]  --conf spark.driver.host=219.219.220.149 --conf spark.driver.cores=2 --conf spark.driver.maxResultSize=2g --conf spark.storage.memoryFraction=0.7  --conf spark.akka.threads=2 --conf spark.akka.frameSize=1024 /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar cs-bwamem -bfn 1 -bPSW 1 -sbatch 10 -bPSWJNI 1  -oChoice 2 -oPath /xubo/alignment/output/g38L100c50Nhs20upload2.adam -localRef 1  -isSWExtBatched 1  0 /home/hadoop/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/GRCH38BWAindex/GRCH38chr1L3556522.fasta  /xubo/alignment/cs-bwamem/input/fastq/g38L100c50Nhs20upload2.fastq
command: cs-bwamem
Map('isPSWJNI -> 1, 'localRef -> 1, 'batchedFolderNum -> 1, 'isPSWBatched -> 1, 'subBatchSize -> 10, 'inFASTQPath -> /xubo/alignment/cs-bwamem/input/fastq/g38L100c50Nhs20upload2.fastq, 'inFASTAPath -> /home/hadoop/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/GRCH38BWAindex/GRCH38chr1L3556522.fasta, 'outputPath -> /xubo/alignment/output/g38L100c50Nhs20upload2.adam, 'isSWExtBatched -> 1, 'isPairEnd -> 0, 'outputChoice -> 2)
CS- BWAMEM command line arguments: false /home/hadoop/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/GRCH38BWAindex/GRCH38chr1L3556522.fasta /xubo/alignment/cs-bwamem/input/fastq/g38L100c50Nhs20upload2.fastq 1 true 10 true ./target/jniNative.so 2 /xubo/alignment/output/g38L100c50Nhs20upload2.adam
HDFS master: hdfs://Master:9000
Input HDFS folder number: 1
Head line: @RG  ID:foo  SM:bar
Read Group ID: foo
Load Index Files
Load BWA-MEM options
Output choice: 2
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[WARNING] Avro: Invalid default for field comment: null not a "bytes"
[WARNING] Avro: Invalid default for field comment: null not a "bytes"
[WARNING] Avro: Invalid default for field comment: null not a "bytes"
CS-BWAMEM Finished!!!
Jun 9, 2016 4:30:59 PM INFO: parquet.hadoop.ParquetInputFormat: Total input paths to process : 1
Jun 9, 2016 4:31:01 PM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
Jun 9, 2016 4:31:02 PM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 49 records.
Jun 9, 2016 4:31:02 PM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
Jun 9, 2016 4:31:02 PM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 99 ms. row count = 49

3.merge：有问题 未解决

hadoop@Master:~/xubo/project/alignment/CountAlignment$ spark-submit --executor-memory 4g --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 4 --master spark://219.219.220.149:7077  --conf spark.driver.host=219.219.220.149 --conf spark.driver.cores=4 --conf spark.driver.maxResultSize=6g --conf spark.storage.memoryFraction=0.7  --conf spark.akka.threads=2 --conf spark.akka.frameSize=1024 /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar merge hdfs://219.219.220.149:9000 /xubo/alignment/output/g38L100c50Nhs20upload2.adam /xubo/alignment/output/g38L100c50Nhs20upload2.merge.adam
command: merge
Total number of new file partitions0
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Number of partitions (0) must be positive.
    at scala.Predef$.require(Predef.scala:233)
    at org.apache.spark.rdd.CoalescedRDD.<init>(CoalescedRDD.scala:81)
    at org.apache.spark.rdd.RDD$$anonfun$coalesce$1.apply(RDD.scala:409)
		at org.apache.spark.rdd.RDD$$anonfun$coalesce$1.apply(RDD.scala:391)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.RDD.coalesce(RDD.scala:390)
    at cs.ucla.edu.bwaspark.dnaseq.MergeADAMFiles$.apply(MergeADAMFiles.scala:41)
    at cs.ucla.edu.bwaspark.BWAMEMSpark$.main(BWAMEMSpark.scala:323)
    at cs.ucla.edu.bwaspark.BWAMEMSpark.main(BWAMEMSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Jun 9, 2016 4:33:23 PM INFO: parquet.hadoop.ParquetInputFormat: Total input paths to process : 1
Jun 9, 2016 4:33:40 PM INFO: parquet.hadoop.ParquetInputFormat: Total input paths to process : 1
hadoop@Master:~/xubo/project/alignment/CountAlignment$

4.读取;
由于没有合并，只能读取partition下面的数据：

package org.gcdss.cli.alignment

import org.gcdss.cli.callVariant.callVariantByAvocado
import org.scalatest.FunSuite

/**
  * Created by xubo on 2016/6/9.
  */
class CountAlignmentSuite extends FunSuite {
  test("artificial fa By runLocal") {
    //    val iString = new SimpleDateFormat("yyyyMMddHHmmssSSS").format(new Date())
    println("start Suite:")
    //    val file = "hdfs://219.219.220.149:9000/xubo/callVariant/avocado/artificial/artificial.realigned.sam"
    val file = "hdfs://219.219.220.149:9000/xubo/alignment/output/g38L100c50Nhs20upload2.adam/0"
    CountAlignment.runLocal(Array(file))

    println("end Suite")
    //      .runLocal
  }
}

运行结果：

start Suite:
start runLocal:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/1win7/java/otherJar/spark-assembly-1.5.2-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/1win7/java/apache-maven-3.3.9-bin/Repository/org/slf4j/slf4j-log4j12/1.7.12/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2016-06-09 16:43:30 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-06-09 16:43:32 WARN  MetricsSystem:71 - Using default name DAGScheduler for source because spark.app.id is not set.
2016-06-09 16:43:34 WARN  :139 - Your hostname, xubo-PC resolves to a loopback/non-reachable address: fe80:0:0:0:482:722f:5976:ce1f%20, but we couldn't find any external IP address!
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2016-06-09 16:43:38 WARN  ParquetRecordReader:193 - Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+--------------------+---------+---------+----+--------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
|              contig|    start|      end|mapq|readName|            sequence|                qual|cigar|basesTrimmedFromStart|basesTrimmedFromEnd|readPaired|properPair|readMapped|mateMapped|firstOfPair|secondOfPair|failedVendorQualityChecks|duplicateRead|readNegativeStrand|mateNegativeStrand|primaryAlignment|secondaryAlignment|supplementaryAlignment|mismatchingPositions|origQual|          attributes|recordGroupName|recordGroupSequencingCenter|recordGroupDescription|recordGroupRunDateEpoch|recordGroupFlowOrder|recordGroupKeySequence|recordGroupLibrary|recordGroupPredictedMedianInsertSize|recordGroupPlatform|recordGroupPlatformUnit|recordGroupSample|mateAlignmentStart|mateAlignmentEnd|mateContig|
+--------------------+---------+---------+----+--------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
|[chr1,248956422,n...| 93465784| 93465884|  26| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|            true|             false|                 false|                 100|    null|NM:i:0    AS:i:100    X...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|101668921|101669021|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|231780491|231780591|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 69447775| 69447875|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 71519060| 71519160|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|187344346|187344446|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|188353053|188353153|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|242097330|242097430|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|112217125|112217225|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|218009808|218009908|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|147407851|147407951|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 45962351| 45962451|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 47126825| 47126925|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|152431792|152431892|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|237080611|237080711|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 82131491| 82131591|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 69486875| 69486975|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|224551051|224551151|   0| chr1-50|TTAAAAAGTCAGGAAAC...|DCDDE<DFEC<CCD,D8...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|              80C9C9|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|154119974|154120074|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 78244128| 78244228|   0| chr1-50|TTCCACAATAGTTGAAC...|@@CDFDFDHFHGHIJH:...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|              9G9G80|    null|NM:i:2    AS:i:90 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
+--------------------+---------+---------+----+--------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
only showing top 20 rows

2016-06-09 16:43:39 WARN  ParquetRecordReader:193 - Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
492
end Suite

参考

【1】https://github.com/xubo245/AdamLearning
【2】https://github.com/bigdatagenomics/adam/ 
【3】https://github.com/xubo245/SparkLearning
【4】http://spark.apache.org
【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job  
【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver

研究成果：

【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)
【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).
【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).
【4】more: https://github.com/xubo245/Publications

Help

If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: xubo245@mail.ustc.edu.cn
Wechat: xu601450868
QQ: 601450868
KeepLearningBigData
关注
专栏目录
mmdetection3d SUN RGB-D数据集预处理
叶子的博客
11-30 5212
SUN RGB-D是普林斯顿大学发布的一种关于室内场景理解的数据集，共包含了10335个样本，其中训练样本和验证测试样本数量分别为5285和5050。每个样本包含了彩色图像（RGB）和深度（D）信息，并且分别进行了二维和三维标注。数据集的具体细节将在后续进行补充，这里主要介绍如何在mmdection3d中使用SUN RGB-D数据集。
【愚公系列】2023年10月 数据结构(四)-栈
最新发布
时光隧道
09-30 4万+
数据结构是计算机科学中的一个重要概念，它描述了数据之间的组织方式和关系，以及对这些数据的访问和操作。常见的数据结构有：数组、链表、栈、队列、哈希表、树、堆和图。数组（Array）：是一种线性数据结构，它将一组具有相同类型的数据元素存储在一起，并为每个元素分配一个唯一的索引。数组的特点是具有随机访问的能力。链表（Linked List）：也是一种线性数据结构，它由一系列的节点组成，每个节点包含数据和指向下一个节点的引用。链表的特点是可以动态地插入或删除节点，但访问某个节点时需要从头开始遍历。
基因数据处理19之BWA匹配算法串产生、匹配、评价等整体流程
Keep Learning
04-15 3077
环境： ubunut BWA samtools wgsim bwa.kit工具 1.数据下载：需要在bwa.kit下 bwa.kit/run-gen-ref hs38DH 2.串产生： hadoop@Mcnode1:~/cloud/adam/xubo/data/hs38DH$ wgsim -N 1000 -1 10 hs38DH.fa hs38DHSE1N10000L
hive的warn
sunlang的博客
06-12 1516
2020-6-12 14:40:24 警告: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl 2020-6-12 14:40:24 信息: parquet.hadoop.Internal.
第62课：SparkSQL下的Parquet使用最佳实践和代码实践学习笔记
slq1023的博客
04-02 5978
第62课：SparkSQL下的Parquet使用最佳实践和代码实践学习笔记 本期内容： 1 SparkSQL下的Parquet使用最佳实践 2 SparkSQL下的Parquet实战   一：Spark SQL下的Parquet使用最佳实践 1， 过去整个业界对大数据的分析的技术栈的Pipeline一般分为以下两种方式： a) Data Source->HDFS->MR/Hive/S
Spark SQL 笔记(18)——spark SQL 总结(1)
一角残叶的博客
11-17 1022
1 Spark SQl 使用场景 Ad-hoc querying of data in files Live SQL analytics over streaming data ETL capabilities alongside familiar SQL Interaction with external Databases Scalable query performance with la...
基因数据处理74之从HDFS读取vcf文件存为Adam的parquet文件（有问题）
Keep Learning
12-25 1107
1.small.vcf: 没记录2.读取：5 load time:3287 ms {"variant": {"variantErrorProbability": 139, "contig": {"contigName": "1", "contigLength": null, "contigMD5": null, "referenceURL": null, "assembly": null, "sp
可用于3D语义分割的RGB-D数据集和点云数据集汇总
qq_34622997的博客
11-05 3603
一、室内RGB-D数据集 1、NYUDv2数据集 https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html 2、SUN-RGBD数据集 http://rgbd.cs.princeton.edu/ SUN-RGBD数据集由四种不同的传感器获取，分别为kinect1,kinect2,xction, SUNRGBD V1包含10335张...
【图像处理算法常用数据集】整理第一弹
qq_45790998的博客
01-14 3955
图像处理数据集，Berkeley Segmentation Dataset (BSDS)、Set14、Urban 100、Kodak dataset、CBSD68、DIV2K 数据集介绍、数据集来源、下载地址、来源文章、来源机构
【测评】海康萤石互联网摄像机增强夜视版测评（型号：CS-C1HC-1D2WFR）
喝酸奶舔盖斯基的专栏
02-28 1万+
目录 1. 开 箱 2. 产品基本参数 3. 功能使用体验 4. 推荐使用场景 5. 进阶推荐 最近入手了（其实是公司年会送的小礼品）一款入门级的海康萤石互联网摄像机（型号：CS-C1HC-1D2WFR），作为一名测试行业深耕多年，拥有国家信息产业部认证的专业评测师，面对有趣的事物情不自禁地想站在职业的角度为大家进行测评一下。 1. 开 箱 产品清单包括1个萤石摄像机、1个电源适配器、1根约3m长的电源线，1个钢板底座及2个泡棉胶粘盘及1份产品使用说明书。将摄像头固定在合适的位置，并按照..
hive任务报错SELECT DISTINCT ‘org.apache.hadoop.hive.metastore.model.MPartition‘
2020xyz的博客
04-30 4912
问题描述： hive执行INSERT OVERWRITE TABLE操作时在最后阶段任务出现报错： FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.
SparkContext
Poor - Because you have no ambition
02-13 793
from pyspark import SparkContext sc = SparkContext() data = sc.parallelize([('amber',22),('alfred',23),('skye',4),('albert',12),('amber',9)]) data ParallelCollectionRDD[0] at parallelize at PythonR...
解决错误requirement failed: numRecords must not be negative
杨先森的博客
05-28 5374
记录spark程序启动时报错修复过程 1、问题： spark程序挂掉几天后重启报错如下信息java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative 2、解决方案 未尝试方案1，结合环境用方案2解决 （方案1）在网上查询有说报错的原因是：删除已经使用过的kafka topic，然后新建同名topic。可以通过把topic名字换一下解决 （方案2）由于本环境offset是存储在hbase里的，有
解决spark任务报错:java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative
a1142939032的博客
03-08 3604
如题，这个任务的报错是有由于消费方的消费条数和注册中心的不一致导致的。解决方法仅供参考哈： 1、先到kafka集群查看消费偏移量 ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list [集群地址:对应端口] --topic [topic名] （Pid(分区id)、Offset(当前已消费的条数)、LogSize(总条数)、Lag(未消费的条数)） 2、修改注册中心zookeeper 的偏移量 a、先进入zookeeper ...
Hive Metastore Server 迁移后hive任务执行失败
qq_35952353的博客
07-22 612
Hive Metastore Server 迁移后hive任务执行失败 启动过程如下： Query ID = root_20190717154242_27560818-a11a-45fa-9ec2-1c524a237169 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated ...
基因数据处理41之mango使用失败
Keep Learning
05-30 1149
这里写代码片
sparksql限制一（多表join问题）
cmoaai42662的博客
03-10 1174
1、sqlContext中3个表不能同时join，需要两两join后再join一次；---经验证，此版本没这个限制（cdh5.3） scala> sqlContext.sql("select a.name,a.age,b...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
weixin_30263277的博客
12-19 8118
控制台包空指针后跟这个异常，是因为控制层调用service时的失败，无法读到sql,问题在于controller在引入的service没有自动装配，在引入多个service时，每个service都要自动装配，不能只有一个@autowired后跟多个service。每一个service都要@autowired。 转载于:https://www.cnblogs.com/ckxlovejava/p/61...
Spark_Spark分区相关的几个算子_partitionBy, coalesce, repartition
迎难而上
06-10 825
Base Spark 2.2.0 首先，我们看下 coalesce 与 repartition 的主要区别： 这两个算子主要区别，在于是否有shuffle 流程的发生。其中 coalesce 没有 shuffle 流程的发生，而 repartition 存在 shuffle。 coalesce rdd 的 coalesce实现 /** * Return a new RDD that is reduced into `numPartitions` partitions. ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值