Tachyon学习笔记-安装部署与运行实例

Tachyon版本：tachyon-0.5.0-bin.tar.gz

[hadoop@centos software]$ tar -xzvf  tachyon-0.5.0-bin.tar.gz

修改/etc/profile文件，内容如下：

TACHYON_HOME=/home/hadoop/software/tachyon-0.5.0

PATH=$PATH:$TACHYON_HOME/bin

export PATH TACHYON_HOME

[hadoop@centos tachyon-0.5.0]$ cp conf/tachyon-env.sh.template conf/tachyon-env.sh

本地化部署

修改tachyon-env.sh文件，添加以下内容：

export JAVA_HOME=/usr/software/jdk

export TACHYON_MASTER_ADDRESS=centos.host1
export TACHYON_UNDERFS_ADDRESS=$TACHYON_HOME/underfs
export TACHYON_WORKER_MEMORY_SIZE=2GB
export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

格式化：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon format

启动Tachyon：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon-start.sh all Mount

浏览器中输入网址：http://centos.host1:19999 查看是否启动正常

运行测试用例：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon runTests

浏览器中可以看到运行结果

分布式部署

Hadoop版本：Hadoop-2.4.1

Tachyon 0.5.0 版本默认支持的Hadoop版本是1.0.4，需要重新编译

[hadoop@centos tachyon-0.5.0]$mvn -Dhadoop.version=2.4.1 -DskipTests -X -U clean package

修改Hadoop配置文件core-site.xml，添加以下内容：

<property>

	<name>fs.tachyon.impl</name>

	<value>tachyon.hadoop.TFS</value>

</property>

更新配置：

[hadoop@centos ~]$hadoop dfsadmin -refreshNodes

修改tachyon-env.sh文件，添加以下内容：

export JAVA_HOME=/usr/software/jdk

export TACHYON_MASTER_ADDRESS=centos.host1
export TACHYON_UNDERFS_ADDRESS=hdfs://centos.host1:9000
export TACHYON_WORKER_MEMORY_SIZE=2GB
export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

修改slaves文件，添加worker主机

格式化：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon format

启动Tachyon：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon-start.sh all Mount

[hadoop@centos tachyon-0.5.0]$ jps

3434 TachyonMaster
3570 TachyonWorker

浏览器中输入网址：http://centos.host1:19999 查看是否启动正常

运行测试用例：

[hadoop@centos tachyon-0.5.0]$ bin/tachyon runTests

浏览器中可以看到运行结果

Tachyon常用命令：

创建文件夹

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs mkdir /test/word/input
Successfully created directory /test/word/input

创建文件

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs touch /test/word/input/word.txt
/test/word/input/word.txt has been created

拷贝本地文件到Tachyon

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs copyFromLocal /home/hadoop/temp/user.txt /test/word/input/user.txt
Copied /home/hadoop/temp/user.txt to /test/word/input/user.txt

拷贝Tachyon文件到本地

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs copyToLocal /test/word/input/user.txt /home/hadoop/temp/u.txt
Copied /test/word/input/user.txt to /home/hadoop/temp/u.txt

查看目录

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs ls /test/word/input/
0.00 B    11-21-2014 06:53:46:860  In Memory      /test/word/input/word.txt
69.00 B   11-21-2014 06:57:11:801  In Memory      /test/word/input/user.txt

查看文件信息

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs fileinfo /test/word/input/user.txt
/test/word/input/user.txt with file id 20 have following blocks: 
ClientBlockInfo(blockId:21474836480, offset:0, length:69, locations:[NetAddress(mHost:centos.host1, mPort:29998)])

查看文件

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs cat /test/word/input/user.txt

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs tail /test/word/input/user.txt

移动文件

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs mv /test/word/input/word.txt /test/word/word.txt
Renamed /test/word/input/word.txt to /test/word/word.txt

删除文件

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs rm /test/word/input/user.txt
/test/word/input/user.txt has been removed

统计文件目录下文件数

[hadoop@centos tachyon-0.5.0]$ bin/tachyon tfs count /test/word/input/

加载HDFS文件到Tachyon

[hadoop@centos tachyon-0.5.0]$ bin/tachyon loadufs tachyon://centos.host1:19998/test/word/input/ hdfs://centos.host1:9000/user/hadoop/data/temp/user.json

加载本地文件到Tachyon

[hadoop@centos tachyon-0.5.0]$ bin/tachyon loadufs tachyon://centos.host1:19998/test/word/input/ /home/hadoop/temp/test.txt 

运行WordCount测试用例：

[hadoop@centos ~]$hadoop jar temp/wordcount.jar org.project.modules.hadoop.mapreduce.WordCountMR -libjars /home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar tachyon://centos.host1:19998/user/hadoop/data/temp/word/input tachyon://centos.host1:19998/user/hadoop/data/temp/word/output

输出结果可以在Tachyon上查看

[hadoop@centos ~]$hadoop jar temp/wordcount.jar org.project.modules.hadoop.mapreduce.WordCountMR -libjars /home/hadoop/software/tachyon-0.5.0/core/target/tachyon-0.5.0-jar-with-dependencies.jar tachyon://centos.host1:19998/user/hadoop/data/temp/word/input 

/user/hadoop/data/temp/word/output

输出结果可以在HDFS上查看