leveldb简介与java实现demo

简介

1.简介

Leveldb是一个google实现的非常高效的kv数据库,目前的版本1.2能够支持billion级别的数据量了。 在这个数量级别下还有着非常高的性能,主要归功于它的良好的设计。特别是LSM算法。

2特点

LevelDB 是单进程的服务,性能非常之高,在一台4核Q6600的CPU机器上,每秒钟写数据超过40w,而随机读的性能每秒钟超过10w。 此处随机读是完全命中内存的速度,如果是不命中 速度大大下降

3局限

LevelDB 只是一个 C/C++ 编程语言的库, 不包含网络服务封装, 所以无法像一般意义的存储服务器(如 MySQL)那样, 用客户端来连接它. LevelDB 自己也声明, 使用者应该封装自己的网络服务器.

4实现介绍

leveldb中有一系列参数会与读写的效率有关,将相关的配置以及编译常量统一修改成可运行时配置参数,测试选取最佳配置值。

应用场景

  1. 内存占用大的服务
  • leveldb是持久化型数据库
  1. 写多于读
  • 写操作要大大快于读操作,而顺序读写操作则大大快于随机读写操作。官方网站报道其随机写性能达到40万条记录每秒,而随机读性能达到6万条记录每秒

涉及项目

  • ActiveMQ

限制和弊端

  1. key和value数据尺寸不能太大,在KB级别,如果存储较大的key或者value,将对leveld的读写性能都有较大的影响。每个key所对应的内容不宜太大,超过32KB性能就会下降很快
  2. 一次只允许一个进程访问一个特定的数据库
  3. 没有内置的C/S架构,但开发者可以使用LevelDB库自己封装一个server,不具备“分布式”集群架构能力
  • 百度开源的分布式文件系统BFS(开源地址:https://github.com/baidu/bfs)提 供了mount工具,可以将整个分布式文件系统直接挂载到本地目录,从而可以像操作本地文件一样来操作分布式文件系统中的文件,我们可以利用分布式文件系统本身提供的数据高可靠特性来保证leveldb中数据的安全。
  1. 不支持索引
  2. 不支持事务

java调用

原生leveldb是基于C++开发,java语言无法直接使用;iq80对leveldb使用JAVA 语言进行了“逐句”重开发,经过很多大型项目的验证(比如ActiveMQ),iq80开发的JAVA版leveldb在性能上损失极少(10%)。对于JAVA开发人员来说,我们直接使用即可,无需额外的安装其他lib。

java maven依赖

<dependency>
	<groupId>org.iq80.leveldb</groupId>
	<artifactId>leveldb</artifactId>
	<version>0.7</version>
</dependency>
<dependency>
	<groupId>org.iq80.leveldb</groupId>
	<artifactId>leveldb-api</artifactId>
	<version>0.7</version>
</dependency>

备注:最新版0.10, 2017年12月

测试demo

package cn.demo.leveldb;

import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Map;

import org.iq80.leveldb.DB;
import org.iq80.leveldb.DBFactory;
import org.iq80.leveldb.DBIterator;
import org.iq80.leveldb.Options;
import org.iq80.leveldb.ReadOptions;
import org.iq80.leveldb.Snapshot;
import org.iq80.leveldb.WriteBatch;
import org.iq80.leveldb.WriteOptions;
import org.iq80.leveldb.impl.Iq80DBFactory;
import org.junit.Test;

/**
 * 各接口测试
 * @author lym
 *
 */
public class LevelDBDemoTest {

	private static final String PATH = "/data/leveldb";
	private static final Charset CHARSET = Charset.forName("utf-8");
	private static final File FILE = new File(PATH);
	
	@Test
	public void putTest() {
		DBFactory factory = new Iq80DBFactory();
		// 默认如果没有则创建
		Options options = new Options();
		File file = new File(PATH);
		DB db = null;
		try {
			db = factory.open(file, options);
			byte[] keyByte1 = "key-01".getBytes(CHARSET);
			byte[] keyByte2 = "key-02".getBytes(CHARSET);
			// 会写入磁盘中
			db.put(keyByte1, "value-01".getBytes(CHARSET));
			db.put(keyByte2, "value-02".getBytes(CHARSET));
			String value1 = new String(db.get(keyByte1), CHARSET);
			System.out.println(value1);
			System.out.println(new String(db.get(keyByte2), CHARSET));
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void readFromSnapshotTest() {
		DBFactory factory = new Iq80DBFactory();
		File file = new File(PATH);
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(file, options);
			// 读取当前快照,重启服务仍能读取,说明快照持久化至磁盘,
			Snapshot snapshot = db.getSnapshot();
			// 读取操作
			ReadOptions readOptions = new ReadOptions();
			// 遍历中swap出来的数据,不应该保存在memtable中。
			readOptions.fillCache(false);
			// 默认snapshot为当前
			readOptions.snapshot(snapshot);
			
			DBIterator it = db.iterator(readOptions);
			while (it.hasNext()) {
				Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
						.next();
				String key = new String(entry.getKey(), CHARSET);
				String value = new String(entry.getValue(), CHARSET);
				System.out.println("key: " + key + " value: " + value);
				if (key.equals("key-01")) {
					System.out.println("".equals(value));
				}
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void snapshotTest() {
		DBFactory factory = new Iq80DBFactory();
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(FILE, options);
			
			db.put("key-04".getBytes(CHARSET), "value-04".getBytes(CHARSET));
			// 只能之前到getSnapshot之前put的值,之后的无法获取,即读取期间数据的变更,不会反应出来
			Snapshot snapshot = db.getSnapshot();
			db.put("key-05".getBytes(CHARSET), "value-05".getBytes(CHARSET));
			ReadOptions readOptions = new ReadOptions();
			readOptions.fillCache(false);
			readOptions.snapshot(snapshot);
			DBIterator it = db.iterator(readOptions);
			while (it.hasNext()) {
				Map.Entry<byte[],byte[]> entry = (Map.Entry<byte[],byte[]>) it
						.next();
				String key = new String(entry.getKey(), CHARSET);
				String value = new String(entry.getValue(), CHARSET);
				System.out.println("key: " + key + " value: " + value);
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void writeOptionsTest() {
		DBFactory factory = new Iq80DBFactory();
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(FILE, options);
			WriteOptions writeOptions = new WriteOptions().sync(true);	// 线程安全
			// 没有writeOptions时,会new一个,所以猜测这里添加了这个参数的意义就是可以设置sync和snapshot参数,建议采用这种方式
			db.put("key-06".getBytes(CHARSET), "value-06".getBytes(CHARSET), writeOptions);
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void deleteTest() {
		DBFactory factory = new Iq80DBFactory();
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(FILE, options);
			// 存在会删除,之后查询不出,根据说明可能不是真删除,而是添加一个标记,待测试(大量数据之后删除,文件大小是否明显变化)
			db.delete("key-02".getBytes(CHARSET));
			// 不存在不会报错
			db.delete("key02".getBytes(CHARSET));
			
			Snapshot snapshot = db.getSnapshot();
			ReadOptions readOptions = new ReadOptions();
			readOptions.fillCache(false);
			readOptions.snapshot(snapshot);
			
			DBIterator it = db.iterator(readOptions);
			while (it.hasNext()) {
				Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
						.next();
				String key = new String(entry.getKey(), CHARSET);
				String value = new String(entry.getValue(), CHARSET);
				System.out.println("key: " + key + " value: " + value);
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void writeBatchTest() {
		DBFactory factory = Iq80DBFactory.factory;
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(FILE, options);
			// 批量保存,批量修改
			WriteBatch writeBatch = db.createWriteBatch();
			writeBatch.put("key-07".getBytes(CHARSET), "value-07".getBytes(CHARSET));
			writeBatch.put("key-08".getBytes(CHARSET), "value-08".getBytes(CHARSET));
			writeBatch.put("key-09".getBytes(CHARSET), "value-09".getBytes(CHARSET));
			// 这里也可以添加writeOptions
			db.write(writeBatch);
			
			DBIterator it = db.iterator();
			while (it.hasNext()) {
				Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
						.next();
				String key = new String(entry.getKey(), CHARSET);
				String value = new String(entry.getValue(), CHARSET);
				System.out.println("key: " + key + " value: " + value);
			}
			it.close();
			
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void writeBatchDeleteTest() {
		DBFactory factory = Iq80DBFactory.factory;
		Options options = new Options();
		DB db = null;
		try {
			db = factory.open(FILE, options);
			WriteBatch writeBatch = db.createWriteBatch();
			writeBatch.put("key-10".getBytes(CHARSET), "value-10".getBytes(CHARSET));
			writeBatch.put("key-11".getBytes(CHARSET), "value-11".getBytes(CHARSET));
			// 会将key-01的value置为""
			writeBatch.delete("key-01".getBytes(CHARSET));
			db.write(writeBatch);
			writeBatch.close();
			DBIterator it = db.iterator();
			while (it.hasNext()) {
				Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
						.next();
				String key = new String(entry.getKey(), CHARSET);
				String value = new String(entry.getValue(), CHARSET);
				System.out.println("key: " + key + " value: " + value);
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
}

性能测试

数据量存储随机读一条顺序读一条顺序读所有磁盘占用测试环境速率
10w494ms36ms--174ms9.75Mwindows20.29w/s
40w2772ms46ms1ms551ms28.6Mwindows14.43w/s
100w5533ms44ms1ms457ms71.5Mwindows18.07w/s
1000w117661ms11m+----715Mwindows--

数据类型

1. 对象

需要实现序列化接口

public class DataTest {

	private static final String PATH = "/data/leveldb";
	private static final File FILE = new File(PATH);
	private static final Charset CHARSET = Charset.forName("UTF-8");
	
	@Test
	public void writeObject() {
		Options options = new Options();
		DBFactory factory = Iq80DBFactory.factory;
		DB db = null;
		try {
			db = factory.open(FILE, options);
			User user = new User();
			user.setName("admin");
			user.setSex("男");
			WriteOptions writeOptions = new WriteOptions();
			writeOptions.snapshot(true);
			
			ByteArrayOutputStream baos = new ByteArrayOutputStream();
			ObjectOutputStream oos = new ObjectOutputStream(baos);
			oos.writeObject(user);
			
			db.put(user.getName().getBytes(CHARSET), baos.toByteArray());
		} catch (IOException e) {
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}
	}
	
	@Test
	public void readObject() {
		Options options = new Options();
		DBFactory factory = Iq80DBFactory.factory;
		DB db = null;
		try {
			db = factory.open(FILE, options);
			byte[] valueByte = db.get("admin".getBytes(CHARSET));
			ByteArrayInputStream bais = new ByteArrayInputStream(valueByte);
			ObjectInputStream ois = new ObjectInputStream(bais);
			User user = (User) ois.readObject();
			System.out.println(user);
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (ClassNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} finally {
			if (db != null) {
				try {
					db.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
			}
		}
	}
}

2. 对象集合

@Test
public void writeObjectList() {
	Options options = new Options();
	DBFactory factory = Iq80DBFactory.factory;
	DB db = null;
	try {
		db = factory.open(FILE, options);
		List<User> userList = new ArrayList<User>();
		User user = new User();
		user.setName("admin");
		user.setSex("男");
		User user2 = new User();
		user2.setName("root");
		user2.setSex("女");
		userList.add(user);
		userList.add(user2);
		WriteOptions writeOptions = new WriteOptions();
		writeOptions.snapshot(true);
		
		ByteArrayOutputStream baos = new ByteArrayOutputStream();
		ObjectOutputStream oos = new ObjectOutputStream(baos);
		oos.writeObject(userList);
		
		db.put("userList".getBytes(CHARSET), baos.toByteArray());
	} catch (IOException e) {
		e.printStackTrace();
	} finally {
		if (db != null) {
			try {
				db.close();
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}
}
@Test
public void readObjectList() {
	Options options = new Options();
	DBFactory factory = Iq80DBFactory.factory;
	DB db = null;
	try {
		db = factory.open(FILE, options);
		byte[] valueByte = db.get("userList".getBytes(CHARSET));
		ByteArrayInputStream bais = new ByteArrayInputStream(valueByte);
		ObjectInputStream ois = new ObjectInputStream(bais);
		List<User> userList = new ArrayList<User>();
		userList = (List<User>) ois.readObject();
		System.out.println(userList);
	} catch (IOException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	} catch (ClassNotFoundException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	} finally {
		if (db != null) {
			try {
				db.close();
			} catch (IOException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		}
	}
}

参考:

半小时学会LevelDB原理及应用

161104、NoSQL数据库:key/value型之levelDB介绍及java实现

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
©️2022 CSDN 皮肤主题:我行我“速” 设计师:Amelia_0503 返回首页
评论 2

打赏作者

future_1024

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值