Tachyon版本:tachyon-0.5.0-bin.tar.gz
[hadoop@centos software]$ tar -xzvf tachyon-0.5.0-bin.tar.gz
修改/etc/profile文件,内容如下:
TACHYON_HOME=/home/hadoop/software/tachyon-0.5.0
PATH=$PATH:$TACHYON_HOME/bin
export PATH TACHYON_HOME
[hadoop@centos tachyon-0.5.0]$ cp conf/tachyon-env.sh.template conf/tachyon-env.sh
本地化部署
修改tachyon-env.sh文件,添加以下内容:
export JAVA_HOME=/usr/software/jdk
export TACHYON_MASTER_ADDRESS=centos.host1
export TACHYON_UNDERFS_ADDRESS=$TACHYON_HOME/underfs
export TACHYON_WORKER_MEMORY_SIZE=2GB
export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem
格式化:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon format
启动Tachyon:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon-start.sh all Mount
浏览器中输入网址:http://centos.host1:19999 查看是否启动正常
运行测试用例:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon runTests
浏览器中可以看到运行结果
分布式部署
Hadoop版本:Hadoop-2.4.1
Tachyon 0.5.0 版本默认支持的Hadoop版本是1.0.4,需要重新编译
[hadoop@centos tachyon-0.5.0]$mvn -Dhadoop.version=2.4.1 -DskipTests -X -U clean package
修改Hadoop配置文件core-site.xml,添加以下内容:
<property>
<name>fs.tachyon.impl</name>
<value>tachyon.hadoop.TFS</value>
</property>
更新配置:
[hadoop@centos ~]$hadoop dfsadmin -refreshNodes
修改tachyon-env.sh文件,添加以下内容:
export JAVA_HOME=/usr/software/jdk
export TACHYON_MASTER_ADDRESS=centos.host1
export TACHYON_UNDERFS_ADDRESS=hdfs://centos.host1:9000
export TACHYON_WORKER_MEMORY_SIZE=2GB
export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem
格式化:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon format
启动Tachyon:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon-start.sh all Mount
[hadoop@centos tachyon-0.5.0]$ jps
3434 TachyonMaster
3570 TachyonWorker
浏览器中输入网址:http://centos.host1:19999 查看是否启动正常
运行测试用例:
[hadoop@centos tachyon-0.5.0]$ bin/tachyon runTests
浏览器中可以看到运行结果
Tachyon常用命令:
创建文件夹
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs mkdir /test/word/input
Successfully created directory /test/word/input
创建文件
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs touch /test/word/input/word.txt
/test/word/input/word.txt has been created
拷贝本地文件到Tachyon
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs copyFromLocal /home/hadoop/temp/user.txt /test/word/input/user.txt
Copied /home/hadoop/temp/user.txt to /test/word/input/user.txt
拷贝Tachyon文件到本地
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs copyToLocal /test/word/input/user.txt /home/hadoop/temp/u.txt
Copied /test/word/input/user.txt to /home/hadoop/temp/u.txt
查看目录
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs ls /test/word/input/
0.00 B 11-21-2014 06:53:46:860 In Memory /test/word/input/word.txt
69.00 B 11-21-2014 06:57:11:801 In Memory /test/word/input/user.txt
查看文件信息
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs fileinfo /test/word/input/user.txt
/test/word/input/user.txt with file id 20 have following blocks:
ClientBlockInfo(blockId:21474836480, offset:0, length:69, locations:[NetAddress(mHost:centos.host1, mPort:29998)])
查看文件
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs cat /test/word/input/user.txt
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs tail /test/word/input/user.txt
移动文件
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs mv /test/word/input/word.txt /test/word/word.txt
Renamed /test/word/input/word.txt to /test/word/word.txt
删除文件
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs rm /test/word/input/user.txt
/test/word/input/user.txt has been removed
统计文件目录下文件数
[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs count /test/word/input/
加载HDFS文件到Tachyon
[hadoop@centos tachyon-0.5.0]$ bin/tachyon loadufs tachyon://centos.host1:19998/test/word/input/ hdfs://centos.host1:9000/user/hadoop/data/temp/user.json
加载本地文件到Tachyon
[hadoop@centos tachyon-0.5.0]$ bin/tachyon loadufs tachyon://centos.host1:19998/test/word/input/ /home/hadoop/temp/test.txt
[hadoop@centos ~]$hadoop jar temp/wordcount.jar org.project.modules.hadoop.mapreduce.WordCountMR -libjars /home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar tachyon://centos.host1:19998/user/hadoop/data/temp/word/input tachyon://centos.host1:19998/user/hadoop/data/temp/word/output
输出结果可以在Tachyon上查看
[hadoop@centos ~]$hadoop jar temp/wordcount.jar org.project.modules.hadoop.mapreduce.WordCountMR -libjars /home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar tachyon://centos.host1:19998/user/hadoop/data/temp/word/input
/user/hadoop/data/temp/word/output
输出结果可以在HDFS上查看