LevelDb 资料整理


//z 2014-08-29 09:46:17 L.124'51223 BG57IV3@XCL T161676003 .K.F4029401865[T6,L94,R5,V74]
leveldb介绍

http://code.google.com/p/leveldb/
http://en.wikipedia.org/wiki/LevelDB
http://highscalability.com/blog/2011/8/10/leveldb-fast-and-lightweight-keyvalue-database-from-the-auth.html
http://news.ycombinator.com/item?id=2526032
http://basho.com/blog/technical/2011/07/01/Leveling-the-Field/
http://blog.yufeng.info/archives/1327
http://www.slideshare.net/sunzhidong/google-leveldb-study-discuss

leveldb官方文档
http://leveldb.googlecode.com/svn/trunk/doc/index.html
http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
http://leveldb.googlecode.com/svn/trunk/doc/impl.html
http://leveldb.googlecode.com/svn/trunk/doc/table_format.txt
http://leveldb.googlecode.com/svn/trunk/doc/log_format.txt

leveldb内部实现和源码解析
http://blog.xiaoheshang.info/?cat=26
http://rdc.taobao.com/blog/cs/?p=1378
http://www.cnblogs.com/haippy/archive/2011/12/04/2276064.html
http://blog.csdn.net/sparkliang
http://duanple.blog.163.com/blog/#m=0
http://blog.csdn.net/tankles?viewmode=contents

bigtable/mapreduce/gfs/lsm-tree/skiplist论文
http://blademaster.ixiezi.com/2010/03/27/bigtable:一个分布式的结构化数据存储系统中文版/
http://blademaster.ixiezi.com/2010/03/27/google-mapreduce中文版/
http://blademaster.ixiezi.com/2010/03/27/the-google-file-system中文版/
http://staff.ustc.edu.cn/~jpq/paper/flash/1996-The%20Log-Structured%20Merge-Tree%20%28LSM-Tree%29.pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.9072&rep=rep1&type=pdf

Tair ldb
http://rdc.taobao.com/blog/cs/?p=1394
http://code.taobao.org/p/tair/wiki/index/
http://code.taobao.org/p/tair/src/branches/ldb/src/storage/ldb/
http://www.cnblogs.com/KevinT/category/590804.html
http://www.tuicool.com/topics/11030027

相关资料
http://www.quora.com/What-is-an-SSTable-in-Googles-internal-infrastructure
http://www.ningoo.net/html/tag/dynamo
http://wiki.apache.org/cassandra/MemtableSSTable
http://wiki.apache.org/cassandra/ArchitectureSSTable
http://en.wikipedia.org/wiki/Queuing_theory
http://rdc.taobao.com/team/jm/archives/1344

//z 2014-08-29 09:46:17 L.124'51223 BG57IV3@XCL T161676003 .K.F4029401865[T6,L94,R5,V74]
Notes
leveldb的Write/Delete:

DB::Put/Delete(DB::Open时*dbptr = impl) => DBImpl::Write => (1) 写log: log_->AddRecord (2) 写memtable: WriteBatchInternal::InsertInto(updates, mem_)

leveldb的Get:

DBImpl::Get => (1) 查memtable: mem->Get (2) 查immutable memtable: imm->Get (3) 查文件 versions_->current() => current->Get => Version::Get

leveldb的Compaction:

leveldb在Open/Get/Write时都有可能做Compaction: DB::Open/DBImpl::Get/DBImpl::Write(DBImpl::MakeRoomForWrite) =>DBImpl::MaybeScheduleCompaction => env_->Schedule(&DBImpl::BGWork, this) => (1) 启后台线程 PosixEnv::Schedule (2) DBImpl::BGWork => DBImpl::BackgroundCall => DBImpl::BackgroundCompaction

leveldb的多线程写:

DBImpl::Write的瓶颈在AcquireLoggingResponsibility,多线程写同一个db时互相竞争logger_,性能反而没有单写线程快. 所以为了scale,对leveldb做sharding,将key做hash后分到多个db,这样多线程读写不会相互竞争,经测试 num_threads : num_dbs为1:1时性能最好,充分利用多核

leveldb的性能调优:

通过sharding/batch writes/increase block_size(size per data block, default 4KB)/increase block_cache(LRUCache, default 8MB)/increase write_buffer_size(memtable size, default 4MB)来提高性能,经过测试,单机24-core采用16 threads/16 shards/1000 batch_sizes/block_size 8K/write_buffer_size 32MB能达到70w+ ops/sec的写性能

//z 2014-08-29 09:46:17 L.124'51223 BG57IV3@XCL T161676003 .K.F4029401865[T6,L94,R5,V74]

转载于:https://www.cnblogs.com/IS2120/p/6745634.html

  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值