Tachyon版本:tachyon-0.5.0-bin.tar.gz
Spark版本:spark-1.1.0-bin-hadoop2.4.tgz
Tachyon的安装部署可以参考:Tachyon学习笔记-安装部署与运行实例
修改spark-env.sh文件,内容如下:
SPARK_CLASSPATH=/home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar
启动Spark Shell
[hadoop@centos spark-1.1.0-bin-hadoop2.4]$ master=spark://centos.host1:7077 ./bin/spark-shell
scala> val file = sc.textFile("tachyon://centos.host1:19998/test/word/input/test.txt")
scala> file.foreach(println)
scala> file.saveAsTextFile("tachyon://centos.host1:19998/test/word/input/test1.txt")
运行WordCount测试:
[hadoop@centos spark-1.1.0-bin-hadoop2.4]$bin/spark-submit --class test.SparkWordCount
--master spark://centos.host1:7077 /home/hadoop/temp/sparkwordcount.jar tachyon://centos.host1:19998/test/word/input tachyon://centos.host1:19998/test/word/output
执行结果可以在http://centos.host1:19999/上查看