KFS

预装软件
 - zlib
 - bzip

 - glib
 - python
 - openssl

 - ncurses
 - Boost (preferably, version 1.34 or higher)
 - cmake (preferably, version 2.4.7 or higher)
 - log4cpp (preferably, version 1.0)
 - gcc version 4.3 or higher)
 - xfs-progs devel RPM on Linux
 

一、安装依赖软件
1、安装log4cpp
tar xvzf log4cpp-1.0.tar.gz
cd log4cpp-1.0
./configure
make
make install

2、安装xfs
tar xvzf xfsprogs-3.0.1.tar.gz
cd xfsprogs-3.0.1
./configure
make
make install
make install-dev

3、安装cmake
tar xvzf cmake-2.6.4.tar.gz
cd cmake-2.6.4
./bootstrap
make
make install

4、安装boost
tar xvzf boost_1_39_0.tar.gz
cd boost_1_39_0

5、安装fuse

最后检查ssh localhost命令是否需要密码。 如果没有，执行下列命令：
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

再试一下ssh localhost。如果还没有起作用那么你应该检查你的ssh-agent是不是有什么问题。

二、开始编译
本文假设KFS目录: ~/code/kfs. 为了支持FUSE(fuse version 2.7.3 or higher). 编辑 ~/code/kfs/CMakeLists.txt

SET(Fuse_LIBRARY_DIR "/usr/local/lib")
SET(Fuse_INCLUDE_DIR "/usr/local/include")


1. cd ~/code/kfs
2. mkdir build
3. cd build

选择下列三个其中一个
4. cmake ~/code/kfs/                                                                                       #-- will build DEBUG binaries by default
4. cmake -DCMAKE_BUILD_TYPE:STRING="Release" ~/code/kfs           #-- will build Release binaries by default
4. cmake -DCMAKE_BUILD_TYPE:STRING="RelWithDebInfo" ~/code/kfs    #-- will build reldbg binaries by default

5. gmake
6. gmake install

生成如下文件夹

Executables will be in: ~/code/kfs/build/bin
Libraries will be in: ~/code/kfs/build/lib


生成JAVA支持:

1. cd ~/code/kfs
2. ant jar

得到如下文件
 * ~/code/kfs/build/classes --- This will contain the Java class files
 * ~/code/kfs/build/kfs.jar --- The jar file containing the Java classes
 
将jar加到 CLASSPATH 环境变量

#export CLASSPATH=${CLASSPATH}:~/code/kfs/build/kfs-[version].jar



生成 Python 支持

1. cd to ~/code/kfs/src/cc/access
2. 编辑 kfs_setup.py ，设置include 路径
       kfsext = Extension('kfs', include_dirs ['~/code/kfs/src/cc/', '<path to boost>'])
3. python kfs_setup.py ~/code/kfs/build/lib/ build          #生成共享库文件_kfs.so_.
4. python kfs_setup.py ~/code/kfs/build/lib/ install      #To install in site-packages for python:

(
安装在其它路径如： ~/code/kfs/build/lib
python kfs_setup.py ~/code/kfs/build/lib install --home=~/code/kfs/build/lib

如果安装在其它路径，更新 PYTHONPATH 、LD_LIBRARY_PATH环境变量:
export PYTHONPATH=${PYTHONPATH}:~/code/kfs/build/lib/lib64/python
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/code/kfs/build/lib
)


开始部署
There are two modes of deployment:
 - Single machine setting: The metaserver/chunkserver run on the same,
 single machine
 - Distributed setting: The servers run on a cluster of machines.


For either type of server, the configuration file defines three
variables:

 * node: This defines the machine name where the binary should run
 * rundir: This defines the directory on the machine where KFS binaries will be installed.
 * baseport: This port at which the metaserver/chunkserver process will listen for connection from clients
 * loglevel: The level for outputting messages. Since KFS uses log4cpp, the values are INFO/DEBUG

For the metaserver, an additional variable is defined:
 * backup_path: This can be used to specify the (remote) location to
 which the metaserver checkpoint files should be backed up to.
 
 * clusterkey : A key that is shared between metaserver/chunkserver.

For the chunkservers, there is are two additional variables that are defined:
  * space: The storage space exported by a chunkserver for storing chunks (units are 'G' for GigaBytes and 'M' for MegaBytes)
  * chunkDir : The list of directories used to store chunk files on the chunkserver nodes.
  For a JBOD configuration, this would be a space separated list of directory names.


By default,
 - meta server's checkpoint/log files are stored in
 ${rundir}/bin/kfscp and ${rundir}/bin/kfslog
 - chunkserver's checkpoint/log files are stored in
 ${rundir}/bin/logs
 - chunkserver's chunks are stored in ${rundir}/bin/kfschunk.  This
 value is overridden when "chunkdir" variable is defined for a chunkserver.

NOTE: It is not advisable to change the default location for either
server's checkpoint/log files.  Changing them adversely affects the
other helper scripts that are provided (such as, backing up the meta
server's logs/checkpoint files, periodically cleaning out old
checkpoint/log files).

两种配置文件格式. 定义所有服务环境的配置文件：machines.cfg:

[metaserver]
node: machine1
clusterkey: kfs-test-cluster
rundir: /mnt/kfs/meta
baseport: 20000
loglevel: INFO
numservers: 2
[chunkserver_defaults]
rundir: /mnt/kfs/chunk
chunkDir: /mnt/kfs/chunk/bin/kfschunk
baseport: 30000
space: 3400 G
loglevel: INFO

例出所有节点的配置文件：machines.txt. (自己手工写)

10.2.3.1
10.2.3.2
10.2.3.3


To install the KFS binaries, perform the following steps:

1. cd ~/code/kfs/scripts
2. 设置好配置文件
3. 运行：
#--all the servers are on a single host --
python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui -s
#--When the servers are on multiple hosts --
python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui


运行KFS

cd ~/code/kfs/scripts
python kfslaunch.py -f machines.cfg -m machines.txt -s

如果想单独关闭某个 chunkserver ，则可以在 chunkserver 中执行以下命令：
sudo scripts/kfsrun.sh -S -c -f bin/ChunkServer.prp
这样只会关闭这个 chunkserver ，而不影响其他 server.

关闭后重新启动： scripts/kfsrun.sh -s -c -f bin/ChunkServer.prp

停止KFS

cd ~/code/kfs/scripts
python kfslaunch.py -f machines.cfg -m machines.txt -S


检查系统状态

cd ~/code/kfs/build/bin/tools
kfsping -m -s <metaserver host> -p <metaserver port>

The KFS package now includes a simple python-based web server that shows the set of servers that are currently connected to the metaserver.
The KFS web server runs on the same machine as the KFS metaserver. The web server's port is metaserver's baseport + 50.
The KFS web server uses a file "all-machines.txt" to track where the chunkservers should be running.
This file should be manually created and placed in ~/code/kfs/webui before deploying KFS binaries.
The format of this file is the name of the chunkserver machines, one entry per line. For example.

10.2.3.1
10.2.3.2
10.2.3.3

When you open the page, http://node1:20050/cluster-view, the web server will return a web page that lists:

where the chunkservers are currently running
where the chunkservers could not be started (due to ssh failures, etc.)
where the chunkservers have failed

添加新chunkservers

It is not necessary to stop the KFS servers to add new chunkservers. Simply update the machines.cfg file and start the servers

删除KFS Binaries

cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -m machines.txt -b ../build/bin -U


通过FUSE挂载KFS

Launch the KFS servers first
cd ~/code/kfs/build/bin

创建kfs.prp文件.the configuration file should have following entries:

metaServer.host = kfs1   #metaserver runs on host
metaServer.port = 20000

mkdir /tmp/kfs-fuse  # create a mount-point
kfs_fuse /tmp/kfs-fuse -f  # force the system to run the process in foreground


性能测试


测试环境:
We provide preliminary performance numbers from a 9-node cluster using KFS version 0.2 code. The 9 nodes were on 3 different racks. The racks were connected by a GigE switch. Each machine had the following configuration:

CPU: Quad-core i386 processor operating at 2400 MHz
Operating system: SunOS 5.10 Generic_127128-11
Memory: 8GB
Disk: 4TB of disk configured as a ZFS volume
Network interface: 1GigE


Write Performance?
For this test, we had a single client write 20GB of data to the servers. Each chunk was replicated 3-way. We used the KfsPerfWriter binary that is in the code repository. This tool opens a file and writes data to it sequentially.

[tests]$ ./KfsPerfWriter -p perf-test-machines.prp -m 20480 -f /perftest.1
Write rate: 412.244 (Mbps)
Write rate: 51.5305 (MBps)


Read Performance?
For this test, we had a single client read 20GB of data to the servers. We used the KfsPerfReader binary that is in the code repository. This tool opens a file and reads data from it sequentially.

[tests]$ ./KfsPerfReader -p perf-test-machines.prp -m 20480 -f /perftest.1
...
Read rate: 833.8 (Mbps)
Read rate: 104.225 (MBps)



KFS fsck 命令

KFS fsck command could be used to check the health of file system.
The command looks at meta server chunk map and collect stats about chunk and their state printing a summary about them in the end. fsck generates a request to meta server to dump its chunkmap.
It then downloads the chunkmap and parses it to check health of file system.

Running fsck (fast mode)?
This mode of operation downloads the chunkmap and does basic checking of chunks and its replicas to verify the health of file system.

cd $KFS_HOME/scripts
python kfsfsck.py -f machines.cfg
where $KFS_HOME would be something like ~code/kosmosfs. machines.cfg is the cluster config file. verbose option also lists chunks with two copies on same rack.

Running KFS with Replica Checker (slow mode)?
KFS fsck could also be used with replicachecker which allows us to find out the file which has missing blocks. In addition to it, we could also instruct it to delete or move such files to /lost+found directories to restore the system back to healthy state. Since replicachecker build namespace, this mode takes little more time than the fast mode of operation. To be able to run fast mode, we would have to make sure replicachecker and kfsshell are compiled.

Running fsck (slow mode)?
cd $KFS_HOME/scripts
python kfsfsck.py -f machines.cfg -r -b ../build -n ./network.def
Where $KFS_HOME would be something like ~code/kosmosfs. machines.cfg is the cluster config file.
../build is the directory which contains executables like "./bin/emulator/replicachecker and ./bin/tools/kfsshell"
network.def contains list of chunk server nodes one per line in the format [chunkserver_ip chunkserver_port chunkserver_rack 0]

[foo@bar]$ head -4 network.def
192.345.1.1 30000 1 0
192.345.9.2 30000 9 0
192.345.4.3 30000 4 0
192.345.1.4 30000 1 0

Validating replica sizes (very slow mode)?
Using --checksize option we could also check to see if sizes of individual replicas match. This operation would ping all chunk servers sequentially to get the size of replica to build the chunk map with replica sizes.