Spark学习笔记-Tachyon运行Spark

Tachyon版本：tachyon-0.5.0-bin.tar.gz

Spark版本：spark-1.1.0-bin-hadoop2.4.tgz

Tachyon的安装部署可以参考：Tachyon学习笔记-安装部署与运行实例

修改spark-env.sh文件，内容如下：

SPARK_CLASSPATH=/home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar

启动Spark Shell

[hadoop@centos spark-1.1.0-bin-hadoop2.4]$ master=spark://centos.host1:7077 ./bin/spark-shell

scala> val file = sc.textFile("tachyon://centos.host1:19998/test/word/input/test.txt")

scala> file.foreach(println)

scala> file.saveAsTextFile("tachyon://centos.host1:19998/test/word/input/test1.txt")

运行WordCount测试：

[hadoop@centos spark-1.1.0-bin-hadoop2.4]$bin/spark-submit --class test.SparkWordCount 

--master spark://centos.host1:7077 /home/hadoop/temp/sparkwordcount.jar tachyon://centos.host1:19998/test/word/input tachyon://centos.host1:19998/test/word/output

执行结果可以在http://centos.host1:19999/上查看