Web search engines are required to process large volumes of data. This entails having a scalable backend storage infrastructure built on commodity hardware (such as, cluster of PCs running Linux). To address this infrastructure need, at Kosmix, we have developed the Kosmos Distributed Filesystem (KFS). We have released KFS as an open-source project under the terms of the Apache 2.0 license. The initial release is KFS version 0.1 and it is currently in “alpha”. The source code as well as pre-built binaries are available for download at the project site

In a nutshell, KFS virtualizes disk storage on a cluster of machines providing a global namespace. Files are striped across nodes in the cluster and are replicated for fault tolerance/availability. KFS consists of a client library that enables user applications to read/write files stored in KFS.

KFS supports the familiar filesystem interfaces/programming model. The functionality of the KFS API is similar to the model exposed by operating systems such as Linux. To illustrate,
• When a file is created, the filename is visible in the global namespace.
• As data is written to a block of a file, it gets flushed out to the set of servers storing that block. Data written to servers can now be read by other processes.
• For writing/reading, a process can seek to any point in the file and read/write from there.
• Files can be opened for writing multiple times.
• Data can be appened to existing files by opening the file for writing in append mode.

When blocks of a file are striped across nodes in the cluster, KFS stores individual blocks of file as files in the underlying file system (such as, XFS on Linux). To guard against disk corruption, checksums are computed on the blocks and verified on each read. If disk corruption is detected by checksum mismatch, the system discards the corrupted block and uses re-replication to recover lost data.

Each file stored in KFS is typically replicated 3-way. Depending on application needs, the degree of replication for files can be changed on-the-fly.

KFS also contains rudimentary support for block rebalancing. To help with better disk utilization across nodes, the system may periodically migrate data from over-utilized to under-utilized nodes.

KFS client library provides support job placement systems. For instance, a job scheduler can determine the location(s) of a byte range within a file and schedule jobs appropriately.

KFS is implemented in C++. In addition to C++ applications, KFS also contains support for Java (via JNI) and Python applications.

To enable a large class applications to evaluate KFS, we have integrated KFS to be the backing store for other open source projects:
• Hadoop: Hadoop is an open-source project that provides a Map/Reduce implementation. It contains a Filesystem API that allows alternate implementations to be used as the backing store. For example, currently, the set of choices for a backing store are Local filesystem, HDFS, S3 infrastructure. As a new alternative to these choices, KFS is integrated with Hadoop using Hadoop’s Filesystem API. This allows existing Hadoop Map/Reduce applications to use KFS seamlessly. That is, by changing some Hadoop configuration parameters, KFS can be used as the backing store. We have submitted the necessary “glue” code to the Hadoop code-base; it will be included in the next Hadoop release.
• Hypertable: Hypertable is an open source project (being developed at Zvents Inc.) that provides a Big-Table interface. KFS is integrated with Hypertable as the backing store.

We are releasing KFS with the intent of providing a useful storage infrastructure software. It is our hope that KFS will meet the storage needs of various projects. We would be happy to work with anyone interested in using KFS. Please try out KFS and give us your feedback of what works, what you would like to see added/possibly contribute to KFS!