Spark生态之Alluxio学习22--saveAsTextFile alluxio后count （有bug）

最新推荐文章于 2022-03-17 20:31:55 发布
最新推荐文章于 2022-03-17 20:31:55 发布
阅读量895 收藏
点赞数
分类专栏： alluxio 文章标签： spark 本地化 数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xubo245/article/details/53190921
版权

更多代码请见：https://github.com/xubo245/SparkLearning

Spark生态之Alluxio学习 版本：alluxio-1.3.0（tachyon），spark-1.5.2,hadoop-2.6.0

0 以下分析是主要是基于spark-1.5.2和alluxio-1.3.0分析的，默认不支持数据本地化，所以会有跨节点传输

1.解释

先加载到mem，然后count

出现问题，D9是block丢失，主要是因为alluxio内存使用问题，每个work会将数据缓存，而不是分布式缓存，导致后面的不经常用的数据丢失

2.代码：

for j in 1 2 3 4 5 6 7 8 9
do
        for i in 1 2 3 4 5 6 7 8 9 10
        do
                echo 'time:'$i
                sh test.sh 6G  alluxio://Master:19998/xubo/alluxio/uniref/'D'$j'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount
        done


done

3.结果：

alluxio-1.3.0

hadoop@Master:~/disk2/xubo/project/alluxio/timeCount$ tail -f run1time201611122335.txt \
> ;
time:1
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.455s
time:2                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.432s
time:3                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.486s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.605s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.874s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.66s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.488s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.391s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.064s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.018s
time:1                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.9s
time:2                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.706s
time:3                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.779s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.809s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.063s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.188s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.882s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:7.166s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.786s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.84s
time:1                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.316s
time:2                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.533s
time:3                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.299s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.381s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:10.477s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.333s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.183s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:10.117s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:5.327s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.247s
time:1                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:24.306s
16/11/12 23:43:00 WARN QueuedThreadPool: 1 threads could not be stopped         
time:2
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:15.898s
time:3                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:5.999s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:6.053s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:14.931s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.422s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.535s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:6.46s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:6.261s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.549s
time:1                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:16.498s
time:2                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:17.061s
time:3                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:6.414s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:7.03s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:6.505s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:6.117s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:6.952s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.545s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.553s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:17.072s
time:1                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:18.648s
time:2                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:18.926s
time:3
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:17.453s
time:4                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:18.81s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:16.726s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:7.699s
time:7                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:8.189s
time:8                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:16.938s
time:9                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:6.611s
time:10                                                                         
input:alluxio://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:6.54s
time:1
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.424s
time:2
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.496s
time:3
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:34.933s
time:4
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.524s
time:5
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.146s
time:6
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:29.543s
time:7
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:41.073s
time:8
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.614s
time:9
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:28.967s
time:10
input:alluxio://Master:19998/xubo/alluxio/uniref/D7Line.fasta   count:5019006   countTime:32.312s
time:1
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:90.415s
time:2
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:58.984s
time:3
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:31.455s
time:4
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:40.781s
time:5                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:34.593s
time:6                                                                          
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:49.076s
time:7
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:29.161s
time:8
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:36.917s
time:9
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:31.651s
time:10
input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:28.131s
time:1

alluxio-0.7.1

hadoop@Master:~/disk2/xubo/project/alluxio/timeCount$ tail -f run1-0.7.1time201611131949.txt 
time:1
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.893s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.882s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.912s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.939s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.877s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.784s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.835s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.835s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:4.01s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D1Line.fasta   count:78295     countTime:3.865s
time:1                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.01s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.107s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.029s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:3.908s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:3.929s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:3.716s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.085s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.016s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:4.071s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D2Line.fasta   count:156590    countTime:3.903s
time:1                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.228s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.168s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.232s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.072s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.254s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.385s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.201s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.381s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.162s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D3Line.fasta   count:313180    countTime:4.18s
time:1                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.288s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.337s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.447s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.312s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.376s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.322s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.446s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.47s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.354s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D4Line.fasta   count:626360    countTime:4.429s
time:1                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:6.288s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.551s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.598s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.623s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:5.115s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.651s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.472s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.659s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.565s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D5Line.fasta   count:1252720   countTime:4.891s
time:1                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.928s
time:2                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.711s
time:3                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.772s
time:4                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.683s
time:5                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.58s
time:6                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.882s
time:7                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.819s
time:8                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:5.149s
time:9                                                                          
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.627s
time:10                                                                         
input:tachyon://Master:19998/xubo/alluxio/uniref/D6Line.fasta   count:2505440   countTime:4.74s

报错

input:alluxio://Master:19998/xubo/alluxio/uniref/D8Line.fasta   count:10038012  countTime:28.131s
time:1
[Stage 0:>                                                        (1 + 16) / 63]16/11/13 00:06:31 ERROR TaskSetManager: Task 43 in stage 0.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 43 in stage 0.0 failed 4 times, most recent failure: Lost task 43.3 in stage 0.0 (TID 22, 219.219.220.131): java.io.IOException: Block 5469372416 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
    at alluxio.TimeCount$.compute(TimeCount.scala:32)
    at alluxio.TimeCount$.main(TimeCount.scala:24)
    at alluxio.TimeCount.main(TimeCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Block 5469372416 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
		at org.apache.spark.scheduler.Task.run(Task.scala:88)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
	time:2
	[Stage 0:=>                                                       (2 + 16) / 63]16/11/13 00:06:41 ERROR TaskSetManager: Task 24 in stage 0.0 failed 4 times; aborting job
	Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 24 in stage 0.0 failed 4 times, most recent failure: Lost task 24.3 in stage 0.0 (TID 23, 219.219.220.180): java.io.IOException: Block 5721030656 is not available in Alluxio
		at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
		at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
		at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
		at alluxio.client.file.FileInStream.close(FileInStream.java:145)
		at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
		at java.io.DataInputStream.read(DataInputStream.java:100)
		at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
		at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
		at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
		at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
    at alluxio.TimeCount$.compute(TimeCount.scala:32)
    at alluxio.TimeCount$.main(TimeCount.scala:24)
    at alluxio.TimeCount.main(TimeCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Block 5721030656 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
		at org.apache.spark.scheduler.Task.run(Task.scala:88)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
	time:3
	[Stage 0:>                                                        (1 + 16) / 63]16/11/13 00:06:52 ERROR TaskSetManager: Task 43 in stage 0.0 failed 4 times; aborting job
	Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 43 in stage 0.0 failed 4 times, most recent failure: Lost task 43.3 in stage 0.0 (TID 23, 219.219.220.131): java.io.IOException: Block 5469372416 is not available in Alluxio
		at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
		at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
		at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
		at alluxio.client.file.FileInStream.close(FileInStream.java:145)
		at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
		at java.io.DataInputStream.read(DataInputStream.java:100)
		at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
		at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
		at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
		at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
    at alluxio.TimeCount$.compute(TimeCount.scala:32)
    at alluxio.TimeCount$.main(TimeCount.scala:24)
    at alluxio.TimeCount.main(TimeCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Block 5469372416 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
		at org.apache.spark.scheduler.Task.run(Task.scala:88)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
	time:4
	[Stage 0:>                                                        (0 + 16) / 63]16/11/13 00:07:02 ERROR TaskSetManager: Task 59 in stage 0.0 failed 4 times; aborting job
	Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 59 in stage 0.0 failed 4 times, most recent failure: Lost task 59.3 in stage 0.0 (TID 22, 219.219.220.233): java.io.IOException: Block 5486149632 is not available in Alluxio
		at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
		at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
		at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
		at alluxio.client.file.FileInStream.close(FileInStream.java:145)
		at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
		at java.io.DataInputStream.read(DataInputStream.java:100)
		at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
		at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
		at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
		at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
    at alluxio.TimeCount$.compute(TimeCount.scala:32)
    at alluxio.TimeCount$.main(TimeCount.scala:24)
    at alluxio.TimeCount.main(TimeCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Block 5486149632 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
		at org.apache.spark.scheduler.Task.run(Task.scala:88)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
	time:5
	[Stage 0:>                                                        (1 + 16) / 63]16/11/13 00:07:13 ERROR TaskSetManager: Task 59 in stage 0.0 failed 4 times; aborting job
	Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 59 in stage 0.0 failed 4 times, most recent failure: Lost task 59.3 in stage 0.0 (TID 23, 219.219.220.131): java.io.IOException: Block 5486149632 is not available in Alluxio
		at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
		at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
		at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
		at alluxio.client.file.FileInStream.close(FileInStream.java:145)
		at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
		at java.io.DataInputStream.read(DataInputStream.java:100)
		at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
		at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
		at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
		at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
		at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
		at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
		at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
		at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
		at scala.Option.foreach(Option.scala:236)
		at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
		at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
		at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1125)
    at alluxio.TimeCount$.compute(TimeCount.scala:32)
    at alluxio.TimeCount$.main(TimeCount.scala:24)
    at alluxio.TimeCount.main(TimeCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Block 5486149632 is not available in Alluxio
    at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
    at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
    at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
    at alluxio.client.file.FileInStream.close(FileInStream.java:145)
    at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
		at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
		at org.apache.spark.scheduler.Task.run(Task.scala:88)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
	time:6
	[Stage 0:=>                                                       (2 + 16) / 63]16/11/13 00:07:24 ERROR TaskSetManager: Task 59 in stage 0.0 failed 4 times; aborting job
	Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 59 in stage 0.0 failed 4 times, most recent failure: Lost task 59.3 in stage 0.0 (TID 23, 219.219.220.131): java.io.IOException: Block 5486149632 is not available in Alluxio
		at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:134)
		at alluxio.client.file.FileInStream.updateBlockInStream(FileInStream.java:519)
		at alluxio.client.file.FileInStream.updateStreams(FileInStream.java:426)
		at alluxio.client.file.FileInStream.close(FileInStream.java:145)
		at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:194)
		at java.io.DataInputStream.read(DataInputStream.java:100)
		at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
		at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
		at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
		at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)
		at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
		at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1553)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
		at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1125)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
		at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)

参考

【1】https://github.com/Alluxio/alluxio
【2】http://www.alluxio.org/
【3】http://spark.apache.org/docs/1.5.2/programming-guide.html
【4】https://github.com/xubo245/SparkLearning
KeepLearningBigData
关注
专栏目录
saveAsTextFile很慢very slow
Learning
04-21 4756
saveAsTextFile very slow 问题表象：通过spark UI观察整个执行阶段开在saveAsTextFile阶段，很多task一直处于运行阶段，感觉很慢，程序是不是卡出了。 初步判断：saveAsTextFile既然是卡在了这，肯定是IO瓶颈吧？以为其他阶段特别快呀 分析： 首先根据以往经验，saveAsTextFile应该是一个非常快速的操作。 查看save的结果，只有1024M左右（为我的耐心点赞，为了看输出结果，
11.2 Spark与Alluxio整合原理与实战
段智华的博客
06-25 4291
l   11.2  Spark与Alluxio整合原理与实战  11.2.1Spark与Alluxio整合原理          Alluxio，以前称为Tachyon，是世界上第一个内存速度虚拟分布式存储系统。它统一数据访问和桥接计算框架和底层存储系统。应用程序只需要连接Alluxio来访问存储在任何底层存储系统中的数据。此外，Alluxio以内存为中心的架构使数据访问速度比现有解决方案更快。在
spark saveAsTextFile too slow
qq_20488317的博客
05-21 2259
saveAsTextFile很慢，一个文件saveASText要15分钟以上，解决思路：1、配置文件：查看spark-submit的参数    如截图，查看一下这些配置是否设置的过小     2、代码（python）df.rdd.map(lambda x: (random.randint(1, 10240), x)).partitionBy(128).map(lambda (r, x): "%s"...
batch normalization的原理和作用_【读书笔记】深入理解TensorFlow架构设计与实现原理（四）...
weixin_39612297的博客
11-25 80
首先一如既往地是我们的约定环节：TF/TB：TensorFlow/TensorBoard；第6章 TB可视化工具6.1 概述TB Web界面的顶部菜单栏列出了7个功能面板，分别用与展示不同的可视化对象：SCALARS面板展示标量值随时间变化的关系图；IMAGES和AUDIO面板分别展示图像和音频数据；GRAPHS面板展示数据流图；DISTRIBUTIONS和HISTOGRAMS面板分别展示向量值的...
Alluxio架构、场景与部分配置参数详解
weixin_46307478的博客
03-17 2890
Alluxio架构、场景与部分配置参数 Alluxio：架构及数据流 - 简书 (jianshu.com) Alluxio-基于内存的虚拟分布式存储系统_机器爱上学习的博客-CSDN博客_alluxio 1 架构 1.1 概述 ​ Alluxio作为大数据和机器学习生态系统中的一个新的数据访问层，配置在任何持久性存储系统(如Amazon S3、Microsoft Azure对象存储、Apache HDFS或OpenStack Swift)和计算框架(如Apache Spark、Presto或Had
数据编排——数据世界中遗失的拼图
Alluxio的博客
06-30 1522
我们正处于数据变革的早期阶段。各个组织和机构竞相构建数据驱动的文化精神，同时也在数据驱动的应用程序上开展了大量创新。这些应用程序重塑了我们生活的许多方面，从我们工作的方式到我们被医学诊疗的方式。然而，数据的价值还远没有得到充分利用，创新的速度还可以大大提高。我们认为这其中遗失的关键拼图就是数据编排层。 为了让应用程序高效地访问数据，应用往往需要重新发明轮子，这阻碍了当前的创新步伐。当一个工程师或科...
xubo245#SparkLearning#Spark生态之Alluxio学习24--分别读取HDFS和Alluxio的数据进行
07-25
1.解释 2.代码： 3.结果：
Alluxio分布式存储系统-其他
06-12
2、兼容Hadoop HDFS的文件系统接口:基于这套接口Hadoop MapReduce和Spark可以使用Alluxio代替HDFS。 3、可插拔的底层存储：Alluxio支持将内存数据持久化到底层存储系统。Alluxio提供了通用接口以简化对接不同的底层...
alluxio：Alluxio，用于云中分析和机器学习的数据编排
02-06
什么是Alluxio （以前称为Tachyon）是一个虚拟的分布式存储系统。 它弥合了计算框架和存储系统之间的鸿沟，使计算应用程序可以通过公共接口连接到众多存储系统。 了解有关更多信息。 Alluxio项目源自加州大学...
藏经阁-Best Practices for Using Alluxio with Spark.pdf
最新发布
08-26
【阿里云】Best Practices for Using Alluxio with Spark Alluxio是一款开源的内存级统一数据层，旨在解决大数据生态系统中的数据访问效率问题。在现代数据处理环境中，多个计算框架和存储系统并存，但它们通常并不...
spark 集群环境开发部署（hadoop，yarn，zookeeper，alluxio，idea开发环境）
10-25
个人总结的集群搭建运行事例环境文档。条理清晰。有spark standalone模式 yarn模式 zookeeper的搭建 还有分布式文件系统hdfs hadoop搭建，内存式文件系统alluxio。开发环境的搭建idea for scala 的配置 ，和打包事例
浅析Alluxio写操作的四种不同方法
Alluxio的博客
03-01 1082
Alluxio是一个能够数据分析和人工智能工作负载的开源数据编排系统。像Apache Spark或Apache Hive这样的分布式应用程序可以通过Alluxio与兼容HDFS的接口访问Alluxio中的数据，而无需修改代码。我们将外部存储（如HDFS或S3）作为底层存储。Alluxio是底层存储系统之上的一个新的存储层，它不仅可以提高原始I/O的性能，还允许应用程序能够灵活地选择读、写和管理文件...
Alluxio数据block移动逻辑分析
a604947740的博客
02-02 444
Alluxio中的数据不是一成不变的，而是随时在调整，根据不同配置和实际情况，任何数据都可能被移动至顶层（内存），或底层（HDD/SSD），亦或被换出Alluxio。本文从日志和代码出发，简要分析Alluxio内部缓存数据移动的逻辑。
Spark生态之Alluxio学习16--将alluxio文件的block大小修改
Keep Learning
11-16 2119
更多代码请见：https://github.com/xubo245/SparkLearningSpark生态之Alluxio学习 版本：alluxio-1.3.0（tachyon），spark-1.5.2,hadoop-2.6.01.alluxio-1.3.01.0 默认大小alluxio-1.3.0的block默认大小为512M，对于我每个节点只有10几个G的小内存来说，而且还需要进行spark运
Spark Streaming 的saveAsTextFiles遇到的坑
weixin_34319374的博客
01-10 1067
使用sparkStreaming消费数据，并使用Dstream的saveAsTextFile保存数据到hdfs中，通过使用这个方法，生成的文件夹存在问题， 代码例子如下： resultRdd.map(x=>x).saveAsTextFiles("hdfs:ip//data/storage/20181010/"+(new Date()))...
alluxio文件读取交互
东方小硕
03-07 1680
文章目录总流程Client - MasterClient - Worker情况一：本地命中 - LocalFilePacketReader读取流程client端worker端情况二：非本地命中的其他读取行为(Netty远程数据传输)client端worker端Worker - Master非本地读取完成后的同步行为 总流程 Client - Master Client - Worker 情况一：本...
spark stream中的dstream对象saveAsTextFiles问题
艾@七的专栏
09-19 8135
利用spark streaming从kafka读取数据，进行流数据的统计分析，最后产生dstream类型的结果集，但是涉及到数据的保存时，遇到了一点小障碍。 我们都知道，spark中普通rdd可以直接只用saveAsTextFile(path)的方式，保存到本地，hdfs中，但是dstream对象没有saveAsTextFile()方法，只有saveAsTextFiles()方法，而且，其参数只
alluxio简单使用
浪子huang
11-19 7939
Alluxio 本文是基于alluxio官网和自己实践整理。 Alluxio版本：1.8.1 CDH 1.15.2 1、介绍 以内存为中心的分布式虚拟存储系统。Alluxio在上层计算框架和底层存储系统之间架起了桥梁，应用层只需要访问Alluxio即可以访问底层对接了的任意存储系统的数据。作者是李浩源，中国人，所以官网 也提供了中文的文档。 [外链图片转存失败,源站可能有防盗链机制,建议将...
Alluxio简介
weixin_46307478的博客
12-12 4581
Alluxio简介 Alluxio概览 - Alluxio v2.6.1 (stable) Documentation 1.介绍 Alluxio 是世界上第一个面向基于云的数据分析和人工智能的开源的数据编排技术。 它为数据驱动型应用和存储系统构建了桥梁, 将数据从存储层移动到距离数据驱动型应用更近的位置从而能够更容易被访问。 这还使得应用程序能够通过一个公共接口连接到许多存储系统。 Alluxio内存至上的层次化架构使得数据的访问速度能比现有方案快几个数量级。 在大数据生态系统中，Alluxio 位于数
Apache Spark 命令注入漏洞（CVE-2022-33891）修复与利用
./bin/spark-shell --conf spark.acls.enable=true 打开连接： http://ip:4040 命令注入漏洞复现： 要复现该漏洞，需要在 URL 中添加恶意命令，例如 ?doAs=`whoami`。可以创建一个 txt 文件来验证漏洞的存在： ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值