从 leveldb 到分布式 kv 存储系统

salyty

已于 2022-05-25 18:41:48 修改

阅读量5.2k

点赞数

分类专栏：储存文章标签：分布式

于 2018-10-17 10:47:01 首次发布

本文链接：https://blog.csdn.net/salyty/article/details/83106237

版权

本文探讨了LevelDB及其增强版RocksDB的特性，包括多线程压缩、优化的存储结构等，并介绍了基于RocksDB的分布式KV存储系统，如360的Pika和阿里巴巴的Tair。Pika采用多线程模型和BlackWidow引擎，而Tair利用一致性哈希和版本控制实现分布式策略和数据一致性。

摘要由CSDN通过智能技术生成

LevelDB

详细原理见 leveldb 原理解析

RocksDB : enhanced LevelDB

https://rocksdb.org.cn/doc.html
RocksDB是从 levelDB 的某个 commit checkout 出来的，进化版

RocksDB is a C++ library providing an embedded key-value store, where keys and values are arbitrary byte streams. It was developed at Facebook based on LevelDB and provides backwards-compatible support for LevelDB APIs.

RocksDB is optimized for Flash with extremely low latencies. RocksDB uses a Log Structured Database Engine for storage, written entirely in C++. A Java version called RocksJava is currently in development. See RocksJava Basics.

RocksDB features highly flexible configuration settings that may be tuned to run on a variety of production environments, including pure memory, Flash, hard disks or HDFS. It supports various compression algorithms and good tools for production support and debugging.

Features

Designed for application servers wanting to store up to a few terabytes of data on locally attached Flash drives or in RAM
Optimized for storing small to medium size key-values on fast storage – flash devices or in-memory
Scales linearly with number of CPUs so that it works well on processors with many cores

RocksDB Features that are not in LevelDB

Performance

Multithread compaction
Multithread memtable inserts
Reduced DB mutex holding
Optimized level-based compaction style and universal compaction style
Prefix bloom filter
Memtable bloom filter
Single bloom filter covering the whole SST file
Write lock optimization
Improved Iter::Prev() performance
Fewer comparator calls during SkipList searches
Allocate memtable memory using huge page.

Features
12. Column Families
13. Transactions and WriteBatchWithIndex
14. Backup and Checkpoints
15. Merge Operators
16. Compaction Filters
17. RocksDB Java
18. Manual Compactions Run in Parallel with Automatic Compactions
19. Persistent Cache
20. Bulk loading
21. Forward Iterators/ Tailing iterator
22. Single delete
23. Delete files in range
24. Pin iterator