Colossus: Successor to the Google File System (GFS)-CSDN博客

本文探讨了Google的下一代集群级文件系统Colossus，作为Google文件系统（GFS）的继任者，Colossus在存储软件架构上进行了重大创新，包括自动分片元数据层、使用Reed-Solomon编码的数据写入方式、客户端驱动的复制和编码等。文章还讨论了Colossus在成本、可用性和灵活性方面的优势。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Colossus is the successor to the Google File System (GFS) as mentioned in the recent paper on Spanner on OSDI 2012. Colossus is also used by spanner to store its tablets. The information about Colossus is slim compared with GFS which is published in the paper on SOSP 2003. There is still some information about Colossus on the Web. Here, I list some of them.

Storage Architecture and Challenges

On Faculty Summit, July 29, 2010, by Andrew Fikes, Principal Engineer.

The slides. Some interesting points:

Storage Software: Colossus
Next-generation cluster-level file system
Automatically sharded metadata layer
Data typically written using Reed-Solomon (1.5x)
Client-driven replication, encoding and replication
Metadata space has enabled availability analyses
Why Reed-Solomon?
Cost. Especially w/ cross cluster replication.
Field data and simulations show improved MTTF
More flexible cost vs. availability choices

GFS: Evolution on Fast-forward

An interview with Google’s Sean Quinlan by the Association for Computer Machinery (ACM).

View the interview.

Some important info:

“We also ended up doing what we call a “multi-cell” approach, which basically made it possible to put multiple GFS masters on top of a pool of chunkservers.”
“We also have something we called Name Spaces, which are just a very static way of partitioning a namespace that people can use to hide all of this from the actual application.” … “a namespace file describes”
“The distributed master certainly allows you to grow file counts, in line with the number of machines you’re willing to throw at it.” … “Our distributed master system that will provide for 1-MB files is essentially a whole new design. That way, we can aim for something on the order of 100 million files per master. You can also have hundreds of masters.”
BitTable “as one of the major adaptations made along the way to help keep GFS viable in the face of rapid and widespread change.”