leveldb资料整理

 

转自:http://hideto.iteye.com/blog/1328921

最近一段时间在学习leveldb的源码,找到了一些相关的资源,用于后续查考和学习使用。

leveldb介绍 
http://code.google.com/p/leveldb/ 
http://en.wikipedia.org/wiki/LevelDB 
http://highscalability.com/blog/2011/8/10/leveldb-fast-and-lightweight-keyvalue-database-from-the-auth.html 
http://news.ycombinator.com/item?id=2526032 
http://basho.com/blog/technical/2011/07/01/Leveling-the-Field/ 
http://blog.yufeng.info/archives/1327 
http://www.slideshare.net/sunzhidong/google-leveldb-study-discuss 

leveldb官方文档 
http://leveldb.googlecode.com/svn/trunk/doc/index.html 
http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html 
http://leveldb.googlecode.com/svn/trunk/doc/impl.html 
http://leveldb.googlecode.com/svn/trunk/doc/table_format.txt 
http://leveldb.googlecode.com/svn/trunk/doc/log_format.txt 

leveldb内部实现和源码解析 
http://blog.xiaoheshang.info/?cat=26 
http://rdc.taobao.com/blog/cs/?p=1378 
http://www.cnblogs.com/haippy/archive/2011/12/04/2276064.html 

bigtable/mapreduce/gfs/lsm-tree/skiplist论文 
http://blademaster.ixiezi.com/2010/03/27/bigtable:一个分布式的结构化数据存储系统中文版/ 
http://blademaster.ixiezi.com/2010/03/27/google-mapreduce中文版/ 
http://blademaster.ixiezi.com/2010/03/27/the-google-file-system中文版/ 
http://staff.ustc.edu.cn/~jpq/paper/flash/1996-The%20Log-Structured%20Merge-Tree%20%28LSM-Tree%29.pdf 
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.9072&rep=rep1&type=pdf 

Tair ldb 
http://rdc.taobao.com/blog/cs/?p=1394 
http://code.taobao.org/p/tair/wiki/index/ 
http://code.taobao.org/p/tair/src/branches/ldb/src/storage/ldb/ 

相关资料 
http://www.quora.com/What-is-an-SSTable-in-Googles-internal-infrastructure 
http://www.ningoo.net/html/tag/dynamo 
http://wiki.apache.org/cassandra/MemtableSSTable 
http://wiki.apache.org/cassandra/ArchitectureSSTable 
http://en.wikipedia.org/wiki/Queuing_theory 
http://rdc.taobao.com/team/jm/archives/1344 

Notes 
leveldb的Write/Delete: 
DB::Put/Delete(DB::Open时*dbptr = impl) => DBImpl::Write => (1) 写log: log_->AddRecord (2) 写memtable: WriteBatchInternal::InsertInto(updates, mem_) 

leveldb的Get: 
DBImpl::Get => (1) 查memtable: mem->Get (2) 查immutable memtable: imm->Get (3) 查文件 versions_->current() => current->Get => Version::Get 

leveldb的Compaction: 
leveldb在Open/Get/Write时都有可能做Compaction: DB::Open/DBImpl::Get/DBImpl::Write(DBImpl::MakeRoomForWrite) =>DBImpl::MaybeScheduleCompaction => env_->Schedule(&DBImpl::BGWork, this) => (1) 启后台线程 PosixEnv::Schedule (2) DBImpl::BGWork => DBImpl::BackgroundCall => DBImpl::BackgroundCompaction 

leveldb的多线程写: 
DBImpl::Write的瓶颈在AcquireLoggingResponsibility,多线程写同一个db时互相竞争logger_,性能反而没有单写线程快. 所以为了scale,对leveldb做sharding,将key做hash后分到多个db,这样多线程读写不会相互竞争,经测试 num_threads : num_dbs为1:1时性能最好,充分利用多核 

leveldb的性能调优: 

通过sharding/batch writes/increase block_size(size per data block, default 4KB)/increase block_cache(LRUCache, default 8MB)/increase write_buffer_size(memtable size, default 4MB)来提高性能,经过测试,单机24-core采用16 threads/16 shards/1000 batch_sizes/block_size 8K/write_buffer_size 32MB能达到70w+ ops/sec的写性能

leveldb实现解析 淘宝-核心系统研发-存储 那岩 neveray@gmail.com 2011-12-13 目录 一、 代码目录结构 ....................................................................... 1 1. doc/ ............................................................................ 1 2. include/leveldb/ ................................................................. 1 3. db/ ............................................................................. 1 4. table/........................................................................... 1 5. port/ ........................................................................... 1 6. util/ ........................................................................... 1 7. helper/memenv/ ................................................................... 1 二、 基本概念 .......................................................................... 1 1. Slice (include/leveldb/slice.h) .................................................. 1 2. Option (include/leveldb/option.h) .............................................. 2 3. Env (include/leveldb/env.h util/env_posix.h) .................................... 3 4. varint (util/coding.h) ........................................................... 3 5. ValueType (db/dbformat.h) ...................................................... 3 6. SequnceNnumber (db/dbformat.h) ................................................. 4 7. user key......................................................................... 4 8. ParsedInternalKey (db/dbformat.h) .............................................. 4 9. InternalKey (db/dbformat.h) ...................................................... 4 10. LookupKey (db/dbformat.h) ........................................................ 4 11. Comparator (include/leveldb/comparator.h util/comparator.cc) .................... 4 12. InternalKeyComparator (db/dbformat.h) ............................................ 5 13. WriteBatch (db/write_batch.cc) ................................................. 5 14. Memtable (db/memtable.cc db/skiplist.h) .......................................... 5 15. Sstable (table/table.cc) ......................................................... 5 16. FileMetaData (db/version_edit.h) ............................................... 5 17. block (table/block.cc) ........................................................... 6 18. BlockHandle(table/format.h) ...................................................... 6 19. FileNumber(db/dbformat.h) ...................................................... 6 20. filename (db/filename.cc) ........................................................ 6 21. level-n (db/version_set.h) ....................................................... 7 22. Compact (db/db_impl.cc db/version_set.cc) ........................................ 7 23. Compaction(db/version_set.cc) .................................................... 7 24. Version (db/version_set.cc) ...................................................... 8 25. VersionSet (db/version_set.cc) ................................................. 9 26. VersionEdit(db/version_edit.cc) ................................................. 10 27. VersionSet::Builder (db/version_set.cc) ......................................... 11 28. Manifest(descriptor)(db/version_set.cc)...................................... 12 29. TableBuilder/BlockBuilder(table/table_builder.cc table/block_builder.cc) ......... 12 30. Iterator (include/leveldb/iterator.h) ........................................... 12 三、 存储结构的格式定义与操作 .......................................................... 12 1. memtable (db/skiplist.h db/memtable) ............................................ 12 2. block of sstable (table/block_builder.cc table/block.cc) ......................... 14 3. sstable (table/table_bulder.cc/table.cc) ........................................ 16 4. block of log (db/log_format.h db/log_writer.cc db/log_reader.cc) .............. 18 5. log (db/log_format.h db/log_writer.cc db/log_reader.cc) ........................ 18 6. cache(util/cache.cc) .......................................................... 19 7. Snapshot (include/leveldb/snapshot.h) ......................................... 19 8. Iterator (include/leveldb/iterator.h) ........................................... 19 四、 主要流程 ......................................................................... 24 1. open ........................................................................... 24 2. put ............................................................................ 25 3. get ............................................................................ 25 4. delete.......................................................................... 26 5. snapshot........................................................................ 26 6. NewIterator ..................................................................... 26 7. compact......................................................................... 26 五、 总结 ............................................................................. 30 1. 设计/实现中的优化 ............................................................... 30 2. 可以做的优化 .................................................................... 31
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值