heiyeluren的blog（黑夜路人的开源世界）

PHPCup.cn - 中国新锐LAMP论坛

[转] KFS,一个克隆GFS的文件系统收藏

KFS（KOSMOS DISTRIBUTED FILE SYSTEM），一个类似GFS、Hadoop中HDFS 的一个开源的分布式文件系统。

PS: google的三大基石 gfs,bigtable,map-reduce 相对应的开源产品 gfs:kfs(据传google创史人的同窗所创)，hdfs(hadoop的子项目) bigtable:hbase(hadoop的子项目)，Hypertable（从hbase项目组分离出去的，用c++实现） map-reduce:hadoop（apache的项目，java实现，目前创史人在yahoo全力打造，已有2000个以上的节点并行计算的规模）

Google两个共同创始人的两个大学同窗（印度人）Anand Rajaraman和Venky Harinarayan，创立的一个新的搜索引擎Kosmix最近捐献了一个克隆GFS的文件系统KFS项目。Hadoop和Hypertable这两个项目也开始支持KFS来做底层的存储。KFS是用C++写的，但是其client支持C++，Java和Python。那么KFS到底有什么特性呢？

支持存储扩充（添加新的chunckserver,系统自动感知）
有效性（复制机制保证文件有效性）
负载平衡（系统周期地检查chunkservers的磁盘利用，并重新平衡chunkservers的磁盘利用，HDFS现在还没有支持）
数据完整性（当要读取数据时检查数据的完整性，如果检验出错使用另外的备份覆盖当前的数据）
支持FUSE（HDFS也有工具支持FUSE）
使用契约（保证Client缓存的数据和文件系统中的文件保持一致性）

HDFS未支持的高级特性：

支持同一文件多次写入和Append，不像HDFS支持一次写入多次读取和不支持Append（最近要增加Append，但是遇到许多问题）。
文件及时有效，当应用程序创建一个文件时,文件名在系统马上有效。不像HDFS文件只当输入流关闭时才在系统中有效，因此，如果应用程序在关闭前出现异常导致没有关闭输入流，数据将会丢失。

官方网站： http://kosmosfs.sourceforge.net/

来自startup的垂直搜索引擎http://www.kosmix.com/的开源项目，又一个开源的类似google mapreduce 的分布式文件系统，可以应用在诸如图片存储、搜索引擎、网格计算、数据挖掘这样需要处理大数据量的网络应用中。与hadoop集成得也比较好，这样可以充分利用了hadoop一些现成的功能，基于C++。

Introduction

Applications that process large volumes of data (such as, search engines, grid computing applications, data mining applications, etc.) require a backend infrastructure for storing data. Such infrastructure is required to support applications whose workload could be characterized as:

Primarily write-once/read-many workloads
Few millions of large files, where each file is on the order of a few tens of MB to a few tens of GB in size
Mostly sequential access

We have developed the Kosmos Distributed File System (KFS), a high performance distributed file system to meet this infrastructure need.

The system consists of 3 components:

Meta-data server : a single meta-data server that provides a global namespace
Block server: Files are split into blocks or chunks and stored on block servers. Blocks are also known as chunk servers. Chunkserver store the chunks as files in the underlying file system (such as, XFS on Linux)
Client library: that provides the file system API to allow applications to interface with KFS. To integrate applications to use KFS, applications will need to be modified and relinked with the KFS client library.

KFS is implemented in C++. It is built using standard system components such as, TCP sockets, aio (for disk I/O), STL, and boost libraries. It has been tested on 64-bit x86 architectures running Linux FC5.

While KFS can be accessed natively from C++ applications, support is also provided for Java applications. JNI glue code is included in the release to allow Java applications to access the KFS client library APIs.

Features

Incremental scalability: New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes.
Availability: Replication is used to provide availability due to chunk server failures. Typically, files are replicated 3-way.
Per file degree of replication: The degree of replication is configurable on a per file basis, with a max. limit of 64.
Re-replication: Whenever the degree of replication for a file drops below the configured amount (such as, due to an extended chunkserver outage), the metaserver forces the block to be re-replicated on the remaining chunk servers. Re-replication is done in the background without overwhelming the system.
Re-balancing: Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes.
Data integrity: To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk.
File writes: The system follows the standard model. When an application creates a file, the filename becomes part of the filesystem namespace. For performance, writes are cached at the KFS client library. Periodically, the cache is flushed and data is pushed out to the chunkservers. Also, applications can force data to be flushed to the chunkservers. In either case, once data is flushed to the server, it is available for reading.
Leases: KFS client library uses caching to improve performance. Leases are used to support cache consistency.
Chunk versioning: Versioning is used to detect stale chunks.
Client side fail-over: The client library is resilient to chunksever failures. During reads, if the client library determines that the chunkserver it is communicating with is unreachable, the client library will fail-over to another chunkserver and continue the read. This fail-over is transparent to the application.
Language support: KFS client library can be accessed from C++, Java, and Python.
FUSE support on Linux: By mounting KFS via FUSE, this support allows existing linux utilities (such as, ls) to interface with KFS.
Tools: A shell binary is included in the set of tools. This allows users to navigate the filesystem tree using utilities such as, cp, ls, mkdir, rmdir, rm, mv. Tools to also monitor the chunk/meta-servers are provided.
Deploy scripts: To simplify launching KFS servers, a set of scripts to (1) install KFS binaries on a set of nodes, (2) start/stop KFS servers on a set of nodes are also provided.
Job placement support: The KFS client library exports an API to determine the location of a byte range of a file. Job placement systems built on top of KFS can leverage this API to schedule jobs appropriately.
Local read optimization: When applications are run on the same nodes as chunkservers, the KFS client library contains an optimization for reading data locally. That is, if the chunk is stored on the same node as the one on which the application is executing, data is read from the local node.

KFS with Hadoop

KFS has been integrated with Hadoop using Hadoop’s filesystem interfaces. This allows existing Hadoop applications to use KFS seamlessly. The integration code has been submitted as a patch to Hadoop-JIRA-1963 (this will enable distribution of the integration code with Hadoop). In addition, the code as well as instructions will also be available for download from the KFS project page shortly. As part of the integration, there is job placement support for Hadoop. That is, the Hadoop Map/Reduce job placement system can schedule jobs on the nodes where the chunks are stored.