Samba TDB 数据库在嵌入式中的使用

最新推荐文章于 2022-12-13 15:40:09 发布

paky_du

最新推荐文章于 2022-12-13 15:40:09 发布

阅读量1.9k

点赞数

分类专栏：开源工作从Linux开始技术分享

本文链接：https://blog.csdn.net/paky_du/article/details/113263202

版权

开源工作从Linux开始同时被 2 个专栏收录

19 篇文章 0 订阅

订阅专栏

技术分享

17 篇文章 0 订阅

订阅专栏

TDB 数据库是Trivial database的缩写。这个东西不一定听过，但是和它相关的Samba应该玩Linux的人都听过。Samba的一些数据就是用TDB存储的。当然了，我们也可以把它用在我们的嵌入式Linux的小型系统中存储我们的数据。

tdb是用key-value的格式存储数据的。我们可以把一个json格式的数据作为value，再给它起个独一无二的名字作为key。把这样的键值对存储到tdb的数据结构里。tdb用的是hash表的结构对每个key做hash处理然后存储在相应的hash链表上。缺省情况下它的hash链表是131个。如下的信息就是一个简单的tdb的数据概况（tdbtool是tdb的一个工具）：

pakydu@SRVECT:~$ tdbtool paky.tdb 
tdb> info
Size of file/data: 69632/7577
Header offset/logical size: 0/69632
Number of records: 14
Incompatible hash: no
Active/supported feature flags: 0x00000000/0x00000001
Robust mutexes locking: no
Smallest/average/largest keys: 7/15/24
Smallest/average/largest data: 4/525/1147
Smallest/average/largest padding: 5/88/241
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 4
Smallest/average/largest free records: 392/14922/57516
Number of hash chains: 131
Smallest/average/largest hash chains: 0/0/1
Number of uncoalesced records: 1
Smallest/average/largest uncoalesced runs: 1/1/1
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 0/11/2/86/0/1/1
tdb>  keys
key 7 bytes: VERSION
key 16 bytes: G_Energy Manager
key 9 bytes: G_default
key 13 bytes: I_users_index
key 13 bytes: I_roles_index
key 22 bytes: U_default.tmpl_profile
key 14 bytes: U_user_profile
key 12 bytes: G_Technician
key 16 bytes: U_user_basicinfo
key 14 bytes: G_System Admin
key 18 bytes: U_oneday_basicinfo
key 24 bytes: G_Non-Technical End User
key 24 bytes: U_system.default_profile
key 18 bytes: G_Monitoring Staff

tdb> show I_users_index        

key 13 bytes
I_users_index
data 11 bytes
[000] 7B 22 75 73 65 72 22 3A  31 7D 00                 {"user": 1}
tdb> show U_user_basicinfo

key 16 bytes
U_user_basicinfo
data 302 bytes
[000] 7B 22 75 73 65 72 6E 61  6D 65 22 3A 22 75 73 65  {"userna me":"use
[010] 72 22 2C 22 70 61 73 73  77 6F 72 64 22 3A 22 31  r","pass word":"1
[020] 61 31 64 63 39 31 63 39  30 37 33 32 35 63 36 39  a1dc91c9 07325c69
[030] 32 37 31 64 64 66 30 63  39 34 34 62 63 37 32 22  271ddf0c 944bc72"
[040] 2C 22 66 75 6C 6C 6E 61  6D 65 22 3A 22 44 65 66  ,"fullna me":"Def
[050] 61 75 6C 74 20 55 73 65  72 22 2C 22 64 65 73 63  ault Use r","desc
[060] 72 69 70 74 69 6F 6E 22  3A 22 54 68 65 20 66 69  ription" :"The fi
[070] 72 73 74 20 64 65 66 61  75 6C 74 20 75 73 65 72  rst defa ult user
[080] 22 2C 22 65 6D 61 69 6C  22 3A 22 75 73 65 72 40  ","email ":"user@
[090] 65 6D 65 72 73 6F 6E 2E  63 6F 6D 22 2C 22 6F 66  emerson. com","of
[0A0] 66 69 63 65 70 68 6F 6E  65 22 3A 22 34 30 34 2D  ficephon e":"404-
[0B0] 35 35 35 2D 30 31 31 31  22 2C 22 73 6D 73 22 3A  555-0111 ","sms":
[0C0] 22 34 30 34 2D 35 35 35  2D 30 31 37 37 22 2C 22  "404-555 -0177","
[0D0] 6C 6F 63 61 6C 65 22 3A  22 65 6E 2D 55 53 22 2C  locale": "en-US",
[0E0] 22 74 65 61 6D 22 3A 22  22 2C 22 72 6F 6C 65 73  "team":" ","roles
[0F0] 22 3A 5B 22 53 79 73 74  65 6D 20 41 64 6D 69 6E  ":["Syst em Admin
[100] 22 5D 2C 22 61 6C 61 72  6D 6F 75 74 70 75 74 6F  "],"alar moutputo
[110] 6E 6C 79 22 3A 22 66 61  6C 73 65 22 2C 22 64 69  nly":"fa lse","di
[120] 73 63 69 70 6C 69 6E 65  22 3A 5B 5D 7D 00        scipline ":[]}
tdb>

对这个tdb的数据库结构有了一个大概的了解后，我们可以根据自己的应用去确定是否使用到自己的产品中。如果你决定要使用它了，那就要开始下载它的源代码，交叉编译，测试。

下载代码：https://www.samba.org/ftp/tdb/

你可以根据下载任意版本使用，关于不同的版本的编译，我是用过1.2，1.3.1 和1.3.11的，不同的版本的交叉编译会有所不同。下面会详细介绍编译的情况。个人不是很推荐使用最新的版本。

2. 交叉编译代码：我的项目多半是arm的平台。所以我的介绍主要是针对arm平台。

编译工具链：这个可以根据项目，使用通用的arm-linux-gnueabihf-gcc，或者芯片厂商提供的。
交叉编译：

version 1.2	version 1.3.1	version 1.3.11
./configure CC=arm-linux-gnueabihf-gcc \ --host=arm-linux Note: 这个版本使用的是比较通用的configure方式，具体的可以使用 ./configure --help 获得更多的参数设置帮助	CC=arm-linux-gnueabihf-gcc ./configure -C --prefix=${D}/usr/ --cross-compile \ --cross-answers=answers.txt	CC=arm-linux-gnueabihf-gcc ./configure --cross-compile --cross-execute='qemu-arm-static -L /usr/arm-linux-gnueabihf'

version 1.2

version 1.3.1

version 1.3.11

./configure CC=arm-linux-gnueabihf-gcc \

--host=arm-linux

Note:

这个版本使用的是比较通用的configure方式，具体的可以使用

./configure --help 获得更多的参数设置帮助

CC=arm-linux-gnueabihf-gcc ./configure -C --prefix=${D}/usr/ --cross-compile \

--cross-answers=answers.txt

CC=arm-linux-gnueabihf-gcc ./configure --cross-compile --cross-execute='qemu-arm-static -L /usr/arm-linux-gnueabihf'

version 1.2的方式很多开源软件都是用的这种方式，这里不做过多介绍。主要对1.3.1和1.3.11介绍一下，这两个版本主要是用的waf的方式：https://wiki.samba.org/index.php/Waf

Note：在这里在强调一个东西：比较新的libtdb使用的是waf，这个是要依赖pythone的，这个要保证你使用的代码别的版本和你的电脑上python的版本匹配。

cross-compiling
You can build for non-host architectures by using the cross-compilation features of the build.

There are two approaches to cross compiling with the waf build:

using a --cross-execute emulator
using a --cross-answers file
In either case you will need a cross-compiler setup installed.

Using --cross-execute
If you have an emulator setup to run cross-compiled binaries then you can use --cross-execute. This will allow you to accurately configure your build with the correct answers.

For example, on a Debian derived system for an ARM target you would install gcc-4.3-arm-linux-gnu, linux-kernel-headers-arm-cross and qemu-arm-static. That will give you enough to create ARM binaries.

You may find the following apt sources useful for this:

deb http://mirror.csclub.uwaterloo.ca/emdebian/ stable main
deb-src http://mirror.csclub.uwaterloo.ca/emdebian/ stable main
With those installed, you would then configure like this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-execute='qemu-arm-static -L /usr/arm-linux-gnu'
that tells the build to use the arm-linux-gnu-gcc compiler, and to cross-execute the test binaries during the configure stage using the qemu-arm-static tool. By using a --cross-execute tool you can ensure that all your configure tests can run correctly, which avoids the problem of having to work out things like the size of various target architecture types.

After the configure, run waf as usual to build. You can then test the resulting binaries using (for example):

qemu-arm-static -L /usr/arm-linux-gnu bin/ldbadd
and it will run using qemu.

Using --cross-answers
If you don't have a emulator for your target machine, you will need to use the --cross-answers option. When you specify something like --cross-answers=target.txt the build will place unknown configure tests in target.txt. You then need to edit that file, and change any UNKNOWN answers with the correct answers for your target.

For example, you could do this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-answers=arm.txt
That will initially create a file like this:

Checking simple C program: UNKNOWN
The configure stopped at that point as it needs to know if a simple C program can run before it will continue. If you look in bin/config.log (or in bin/.conf_check_0/) you will see the source code for the program that it was testing.

To continue the configure process, you would edit arm.txt and replace UNKNOWN with OK like this:

Checking simple C program: OK
The configure will again fail, and you will end up with this in arm.txt:

Checking simple C program: OK
building library support: UNKNOWN
Checking for large file support: UNKNOWN
Checking for -D_FILE_OFFSET_BITS=64: UNKNOWN
Checking for WORDS_BIGENDIAN: UNKNOWN
Checking size of char: UNKNOWN
Checking size of int: UNKNOWN
Checking size of long long: UNKNOWN
Checking size of long: UNKNOWN
Checking size of off_t: UNKNOWN
Checking size of short: UNKNOWN
Checking size of size_t: UNKNOWN
Checking size of ssize_t: UNKNOWN
Checking size of dev_t: UNKNOWN
Checking size of ino_t: UNKNOWN
Checking size of time_t: UNKNOWN
Checking size of void*: UNKNOWN
Checking for C99 vsnprintf: UNKNOWN
Checking for HAVE_SECURE_MKSTEMP: UNKNOWN
Note that only the tests that involve running an executable on the target have been added. The other tests are able to be completed by running on the host, and thus do not need a cross-answers entry.

You then should replace all the UNKNOWN values with the correct values for your platform. For example:

Checking simple C program: OK
building library support: OK
Checking for large file support: OK
Checking for -D_FILE_OFFSET_BITS=64: OK
Checking for WORDS_BIGENDIAN: OK
Checking size of char: "1"
Checking size of int: "4"
Checking size of long long: "8"
Checking size of long: "4"
Checking size of off_t: "8"
Checking size of short: "2"
Checking size of size_t: "4"
Checking size of ssize_t: "4"
Checking size of dev_t: "4"
Checking size of ino_t: "4"
Checking size of time_t: "4"
Checking size of void*: "4"
Checking for C99 vsnprintf: OK
Checking for HAVE_SECURE_MKSTEMP: OK
The right hand side of each line can be in one of several formats, all of which map to a exit code and a returned string. There are also shortcut forms:

a OK means that the test should return a exit code of 0, and an empty string
a FAIL or NO means that the test returns a exit code of 255, and an empty string
a UNKNOWN means the test has an unknown result
a "STRING" or 'STRING' means the test has an exit code of 0, and the given string
The general form is:

Some message: (RETCODE, RETSTRING)
here are some examples:

Some message: (0, "")
Some message: (23, "")
Some message: (0, "a string")
Some message: (1, "another string")
If your configure step completes, but there are still UNKNOWN values in your answers file, then the following message will be displayed:

Cross answers file arm.txt is incomplete
and you will need to continue with the iterative process until all answers are filled in

上面的信息是来自samba官网的编译介绍。两种方式都可以成功。我附上自己目前使用的answer.txt:

Checking uname sysname type:"Linux"
Checking uname machine type: "arm"
Checking uname release type: "5.4.0-58-generic"
Checking uname version type: "#64~18.04.1-Ubuntu SMP Wed Dec 9 17:11:11 UTC 2020"
Checking simple C program: "ok"
rpath library support: "ok"
-Wl,--version-script support: "ok"
Checking getconf LFS_CFLAGS: "-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"
Checking for large file support without additional flags: "ok"
Checking for -D_FILE_OFFSET_BITS=64: "yes"
Checking for -D_LARGE_FILES: "yes"
Checking getconf large file support flags work: "ok"
Checking correct behavior of strtoll: "ok"
Checking for working strptime: "ok"
Checking for C99 vsnprintf: "ok"
Checking for HAVE_SHARED_MMAP: "ok"
Checking for HAVE_MREMAP: "ok"
Checking for HAVE_INCOHERENT_MMAP: "ok"
Checking for HAVE_SECURE_MKSTEMP: "ok"
Checking for HAVE_IFACE_GETIFADDRS: "ok"
Checking for HAVE_IFACE_IFCONF: "ok"
Checking for HAVE_IFACE_IFREQ: "ok"

3. 如果上面的交叉编译的configure都可以成功，那接下来只要make就可以得到相关的library和tools。

libtdb.so.1.3.11 tdbbackup tdbdump tdbrestore tdbtool tdbtorture

一些实用代码的情况我们看一参考代码里面文档libtdb-1.3.11.orig\docs\README或者https://tdb.samba.org/

4. 在最后分享一个tdb文件被损害破坏的问题：

./tdbtool /data/settings/DB/paky.tdb
tdb> info
tdb_rec_read bad magic 0xd9fee666 at offset=4783600
Size of file/data: 23068672/480025
Header offset/logical size: 0/23068672
Number of records: 1966
Incompatible hash: no
Active/supported feature flags: 0x00000000/0x00000001
Robust mutexes locking: no
Smallest/average/largest keys: 2/25/41
Smallest/average/largest data: 3/218/43854
Smallest/average/largest padding: 4/71/10973
Number of dead records: 1
Smallest/average/largest dead records: 7938024/7938024/7938024
Number of free records: 1777
Smallest/average/largest free records: 12/8114/4062036
Number of hash chains: 131
Smallest/average/largest hash chains: 0/14/30
Number of uncoalesced records: 1356
Smallest/average/largest uncoalesced runs: 1/8/92
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 0/2/1/63/34/0/0
tdb> check
Hashes do not match records
Integrity check for the opened database failed.
tdb>

因为某些异常的发生，tdb的文件里的数据库结构被破坏了，我们没法读出里面的数据了。这个问题我的处理方式时去修改libtdb的代码试图去修正里面的异常magic number.确保可以尽量的读出所有可以读出的数据，处理流程如下：

step 1：修改libtdb的代码试图去修正里面的异常magic number
step 2：使用tdbbackup 备份上面修复后的数据库文件
step 3：再次检查备份处理数据库文件，确认完整无误。
step 4：删除原来的数据库文件，把备份数据库文件改名成正式文件。

最后附上step1中的修改patch for version 1.3.11：

From b39c2622581421e7136c328e86af2db9a86de4af Mon Sep 17 00:00:00 2001
From: "Paky.Du" <paky.pc.du@gmail.com>
Date: Thu, 28 Jan 2021 15:29:14 +0800
Subject: [PATCH] fix the tdb database bad-magic when read record.

---
 common/io.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/common/io.c b/common/io.c
index fe47d18..b2cb242 100755
--- a/common/io.c
+++ b/common/io.c
@@ -645,10 +645,25 @@ int tdb_rec_read(struct tdb_context *tdb, tdb_off_t offset, struct tdb_record *r
 	if (tdb->methods->tdb_read(tdb, offset, rec, sizeof(*rec),DOCONV()) == -1)
 		return -1;
 	if (TDB_BAD_MAGIC(rec)) {
+	#if 0
 		/* Ensure ecode is set for log fn. */
 		tdb->ecode = TDB_ERR_CORRUPT;
 		TDB_LOG((tdb, TDB_DEBUG_FATAL,"tdb_rec_read bad magic 0x%x at offset=%u\n", rec->magic, offset));
 		return -1;
+	#else
+		//Try to fix the error...
+		if (rec->magic != TDB_MAGIC) {
+		/* this happens when a app is showdown while deleting a record - we should
+		   not completely fail when this happens */
+		
+			TDB_LOG((tdb, TDB_DEBUG_WARNING, "tdb_rec_read bad magic 0x%x at offset=%u - fixing\n",
+				 rec->magic, offset));
+			//rec->magic = TDB_MAGIC;
+			rec->magic = TDB_DEAD_MAGIC;
+			if (tdb_rec_write(tdb, offset, rec) == -1)
+				return -1;
+		}
+	#endif
 	}
 	return tdb->methods->tdb_oob(tdb, rec->next, sizeof(*rec), 0);
 }
-- 
2.17.1

paky_du

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Samba TDB 数据库在嵌入式中的使用

Samba stores its data in TDB files. TDB stands for "Trivial database"其他相关的数据库的介绍可以参考这个链接：https://www.jianshu.com/p/8bb2c9d8d90c?utm_campaign=maleskine&utm_content=note&utm_medium=seo_notes&utm_source=recommendation...
复制链接

扫一扫