目录
3.1 所有的子线程都在InitServerLast函数中产生
3.3.3 主线程(也即io_thd_0)线程中关于IO模型的相关逻辑
3.3.4 handleClientsWithPendingReadsUsingThreads(类似于Write)
3.3.5 详细叙述handleClientsWithPendingWritesUsingThreads
3.3.6 稍微看下startThreadedIO()#networking.c
3.3.7 阅读readQueryFromClient()#networking.c中与IO多线程中的子线程相关部分
3.3.8 对于IO多线程中的子线程有意义的postponeClientRead#networking.c
3.3.9 阅读readQueryFromClient()#networking.c中与IO多线程中的主线程相关部分
3.3.10 关于IO线程组中的子线程的任务分配参数io-threads-do-reads
3.4.2 bioProcessBackgroundJobs
0.阅读指南
一些链接
蚂蚁金服专家对于io-threads-do-reads参数的讲解
Redis 多线程网络模型全面揭秘-原文链接
关注【io-threads-do-reads yes】这个用法-自测一下
Redis面试题（一）: Redis到底是多线程还是单线程？
LINUX下动态链接库的使用（dlopen/dlsym/dlclose/dlerror）
===================================================================================
Redis之IO线程、IO多路复用，BIO、NIO和AIO区别
多线程（15） pthread_setcancelstate 及 其他线程取消函数
===================================================================================
Redis的碎片整理功能只有在使用jemalloc的时候才支持
Redis 4.0 自动内存碎片整理（Active Defrag）源码分析
相关关键字
dlsym技术
pthread_sigmask
pthread_key_create
pthread_atfork--在malloc_init_hard_recursible函数中
1.一些人说的"Redis是单线程"是指什么？
如截图所示，《Redis深度历险》作者说Redis是单线程的，如果产生怀疑，就是基础知识不足.
我对这句话感觉非常不妥.
什么是单线程？
百度百科中说，单线程就是一个进程只有一个线程.
什么是多线程？
多线程就是一个进程有多个线程.
如果作者一定要说“基础知识不足”伤人自尊心的话，对比着我的Redis-0.091，Redis-3.0.7
和Redis-6.0.9的测试结果,我觉得非常不妥,即使这本书写的6.0版本之前，我觉得把
“Redis的网络模型是单线程”的，而不是Redis是单线程的,这个才是严谨的说法！！
或许，作者的“线程IO模型”就是指的“Redis的网络模型”？
也许每个人说的话的背景是不一样的，但我要说的是，Redis作为一个整体而言，它在很早的
版本中就不是单线程的了，那么很多人都说的单线程究竟体现在哪呢？
不发牢骚了，我继续探究问题吧.看看到底为啥那么多人都说“单线程”，为啥.
等我研究好了，估计也就知道了,
我期望得到的研究结果是：
在XXX版本中XXX模块里redis采用了单线程.
而不是一句“Redis是单线程的”.
通过研究可以得出结论：
redis-server在6.0版本之前的IO模块中采用了单线程，在6*版本之后提供了IO模块多线程的选项，其作为一个
整体对外提供服务的程序，它在很早的版本中就不是单线程程序. 所以虽然写出高水平参考书的老梁说出“Redis
是个单线程程序”，我们也应该知道他的这个说法是非常不严谨的.
2.研究【6.0.8】版本的redis线程
如下，我的测试环境中的从官方下载下的源码编译的没改过任何配置和代码的redis-server中有五个线程，用我的gdb来看看它们究竟是哪些线程吧.
cd /home/muten/module/redis-6.0.8/redis-6.0.8/src
gdb ./redis-server
(gdb) r
(gdb) i threads
Id Target Id Frame
5 Thread 0x7fffecaba700 (LWP 45141) "jemalloc_bg_thd" 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7fffed2bb700 (LWP 45140) "bio_lazy_free" 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7fffedabc700 (LWP 45139) "bio_aof_fsync" 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2 Thread 0x7fffee2bd700 (LWP 45138) "bio_close_file" 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7ffff7fc10c0 (LWP 45134) "redis-server" 0x00007ffff6fb0f43 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff6fb0f43 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043037e in aeApiPoll (tvp=<optimized out>, eventLoop=0x7ffff4c0b480) at ae_epoll.c:112
#2 aeProcessEvents (eventLoop=eventLoop@entry=0x7ffff4c0b480, flags=flags@entry=27) at ae.c:447
#3 0x000000000043078d in aeMain (eventLoop=0x7ffff4c0b480) at ae.c:539
#4 0x000000000042d077 in main (argc=1, argv=0x7fffffffdad8) at server.c:5349
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fffee2bd700 (LWP 45138))]
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000048cc6b in bioProcessBackgroundJobs (arg=0x0) at bio.c:187
#2 0x00007ffff7287ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff6fb096d in clone () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7fffedabc700 (LWP 45139))]
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000048cc6b in bioProcessBackgroundJobs (arg=0x1) at bio.c:187
#2 0x00007ffff7287ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff6fb096d in clone () from /lib64/libc.so.6
(gdb) thread 4
[Switching to thread 4 (Thread 0x7fffed2bb700 (LWP 45140))]
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000048cc6b in bioProcessBackgroundJobs (arg=0x2) at bio.c:187
#2 0x00007ffff7287ea5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff6fb096d in clone () from /lib64/libc.so.6
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffecaba700 (LWP 45141))]
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007ffff728ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004fb7d1 in background_thread_sleep (tsdn=<optimized out>, interval=<optimized out>, info=<optimized out>) at src/background_thread.c:234
#2 background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:309
#3 background_thread0_work (tsd=<optimized out>) at src/background_thread.c:454
#4 background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:492
#5 background_thread_entry () at src/background_thread.c:522
#6 0x00007ffff7287ea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff6fb096d in clone () from /lib64/libc.so.6
3 线程模型探索
3.1 所有的子线程都在InitServerLast函数中产生
void InitServerLast() {
bioInit();
initThreadedIO();
set_jemalloc_bg_thread(server.jemalloc_bg_thread);
server.initial_memory_usage = zmalloc_used_memory();
}
3.2 线程分类
3.3 IO线程
3.3.1 IO线程的产生和说明
IO线程的个数与配置文件中配置的【io-threads】参数相关,
(1)如果这个参数不配置或者配置为1，就不需要另外启动IO线程，主线程一定是一个IO线程,是io_thd0;
(2)如果这个参数配置为n(n>1)，那么就需要另外启动(n-1)个IO线程.
打开配置文件查看：
vim /home/muten/module/redis-6.0.8/redis-6.0.8/redis.conf
/* Initialize the data structures needed for threaded I/O. */
void initThreadedIO(void) {
server.io_threads_active = 0; /* We start with threads not active. */
// 如果配置文件中(如上图)设置的io线程数量为1，则不启动多余的io线程，只使用主线程
// spawn vt.产卵; 引发; 引起; 导致; 造成;
/* Don't spawn any thread if the user selected a single thread:
* we'll handle I/O directly from the main thread. */
if (server.io_threads_num == 1) return;
// server.io_threads_num的个数不能超过IO_THREADS_MAX_NUM(128)个，否则报错并退出程序
if (server.io_threads_num > IO_THREADS_MAX_NUM) {
serverLog(LL_WARNING,"Fatal: too many I/O threads configured. "
"The maximum number is %d.", IO_THREADS_MAX_NUM);
exit(1);
}
/* Spawn and initialize the I/O threads. */
for (int i = 0; i < server.io_threads_num; i++) {
/* Things we do for all the threads including the main thread. */
io_threads_list[i] = listCreate();
if (i == 0) continue; /* Thread 0 is the main thread. */
/* Things we do only for the additional threads. */
pthread_t tid;
pthread_mutex_init(&io_threads_mutex[i],NULL);
io_threads_pending[i] = 0;
pthread_mutex_lock(&io_threads_mutex[i]); /* Thread will be stopped. */
if (pthread_create(&tid,NULL,IOThreadMain,(void*)(long)i) != 0) {
serverLog(LL_WARNING,"Fatal: Can't initialize IO thread.");
exit(1);
}
io_threads[i] = tid;
}
}
如果配置了多个IO线程，就会执行IOThreadMain函数里面的内容.这个将在下文中介绍.
3.3.2 IO多线程开启情况下的子线程的处理逻辑
- 开启空循环扫描 io_threads_pending 数组，如果找到属于当前线程的那个下标在数组中的值不为0则跳出扫描；
- 再次检查当前线程待处理客户端的数量，如果为0，则当前线程停止运行；
- 从 io_threads_list 列表数组中取出当前线程待处理的client的列表，根据 io_threads_op 全局标志位决定对这些 client 做对应的处理，比如 IO_THREADS_OP_READ读操作则调用readQueryFromClient() 函数继续处理，IO_THREADS_OP_WRITE写操作则调用writeToClient(c,0)；
- 处理完毕后，清空 io_threads_list 列表数组中当前线程待处理的 client 的列表，并将 io_threads_pending 对应下标值置为 0，主线程利用该数组即可知道IO线程是否执行完所有读写任务.
void *IOThreadMain(void *myid) {
/* The ID is the thread number (from 0 to server.iothreads_num-1), and is
* used by the thread to just manipulate a single sub-array of clients. */
long id = (unsigned long)myid;
char thdname[16];
snprintf(thdname, sizeof(thdname), "io_thd_%ld", id);
redis_set_thread_title(thdname);
redisSetCpuAffinity(server.server_cpulist);
while(1) {
/* Wait for start */
for (int j = 0; j < 1000000; j++) {
if (io_threads_pending[id] != 0) break;
}
/* Give the main thread a chance to stop this thread. */
if (io_threads_pending[id] == 0) {
pthread_mutex_lock(&io_threads_mutex[id]);
pthread_mutex_unlock(&io_threads_mutex[id]);
continue;
}
serverAssert(io_threads_pending[id] != 0);
if (tio_debug) printf("[%ld] %d to handle\n", id, (int)listLength(io_threads_list[id]));
/* Process: note that the main thread will never touch our list
* before we drop the pending count to 0. */
listIter li;
listNode *ln;
listRewind(io_threads_list[id],&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
if (io_threads_op == IO_THREADS_OP_WRITE) {
writeToClient(c,0);
} else if (io_threads_op == IO_THREADS_OP_READ) {
readQueryFromClient(c->conn);
} else {
serverPanic("io_threads_op value is unknown");
}
}
listEmpty(io_threads_list[id]);
io_threads_pending[id] = 0;
if (tio_debug) printf("[%ld] Done\n", id);
}
}3.3.3 主线程(也即io_thd_0)线程中关于IO模型的相关逻辑
1.main函数中的initServer函数中注册eventLoop->beforesleep函数，将其赋值为beforeSleep;
void initServer(void) {
...
aeSetBeforeSleepProc(server.el,beforeSleep);
...
}
2.main函数中的aeMain的实现：
void aeMain(aeEventLoop *eventLoop) {
eventLoop->stop = 0;
while (!eventLoop->stop) {
aeProcessEvents(eventLoop, AE_ALL_EVENTS|
AE_CALL_BEFORE_SLEEP|
AE_CALL_AFTER_SLEEP);
}
}
3.接着调用 aeProcessEvents，在aeProcessEvents中相关重要实现如下：
int aeProcessEvents(aeEventLoop *eventLoop, int flags)
{
...
if (eventLoop->beforesleep != NULL && flags & AE_CALL_BEFORE_SLEEP)
eventLoop->beforesleep(eventLoop);
...
}
如第一步中所说，eventLoop->beforesleep被赋值为beforeSleep，所以我们接下来要看的是beforeSleep
的实现.
4.beforeSleep函数中与我们的IO模块使用多线程模型的相关重要代码如下：
void beforeSleep(struct aeEventLoop *eventLoop) {
...
handleClientsWithPendingReadsUsingThreads();// 使用IO线程处理等待读取数据的客户端
...
handleClientsWithPendingWritesUsingThreads();// 使用IO线程处理等待响应的客户端
...
}
5.对于beforeSleep函数中的重要逻辑我们会在下面小节中详细叙述.3.3.4 handleClientsWithPendingReadsUsingThreads(类似于Write)
int handleClientsWithPendingReadsUsingThreads(void) {
if (!server.io_threads_active || !server.io_threads_do_reads) return 0;
int processed = listLength(server.clients_pending_read);
if (processed == 0) return 0;
if (tio_debug) printf("%d TOTAL READ pending clients\n", processed);
/* Distribute the clients across N different lists. */
listIter li;
listNode *ln;
listRewind(server.clients_pending_read,&li);
int item_id = 0;
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
int target_id = item_id % server.io_threads_num;
listAddNodeTail(io_threads_list[target_id],c);
item_id++;
}
/* Give the start condition to the waiting threads, by setting the
* start condition atomic var. */
io_threads_op = IO_THREADS_OP_READ;
for (int j = 1; j < server.io_threads_num; j++) {
int count = listLength(io_threads_list[j]);
io_threads_pending[j] = count;
}
/* Also use the main thread to process a slice of clients. */
listRewind(io_threads_list[0],&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
readQueryFromClient(c->conn);
}
listEmpty(io_threads_list[0]);
/* Wait for all the other threads to end their work. */
while(1) {
unsigned long pending = 0;
for (int j = 1; j < server.io_threads_num; j++)
pending += io_threads_pending[j];
if (pending == 0) break;
}
if (tio_debug) printf("I/O READ All threads finshed\n");
/* Run the list of clients again to process the new buffers. */
while(listLength(server.clients_pending_read)) {
ln = listFirst(server.clients_pending_read);
client *c = listNodeValue(ln);
c->flags &= ~CLIENT_PENDING_READ;
listDelNode(server.clients_pending_read,ln);
if (c->flags & CLIENT_PENDING_COMMAND) {
c->flags &= ~CLIENT_PENDING_COMMAND;
if (processCommandAndResetClient(c) == C_ERR) {
/* If the client is no longer valid, we avoid
* processing the client later. So we just go
* to the next. */
continue;
}
}
processInputBuffer(c);
}
/* Update processed count on server */
server.stat_io_reads_processed += processed;
return processed;
}3.3.5 详细叙述handleClientsWithPendingWritesUsingThreads
- (1) 查看数组 server.clients_pending_write 长度是否为0，为 0 则没有在等待响应的客户端，不需要使用 IO 线程处理. Redis使用了server.clients_pending_write全局数组来存放等待响应的客户端，另一个数组 server.clients_pending_read则存放等待读取数据的客户端；
- (2) 如果server.io_threads_num配置的线程数为1或stopThreadedIOIfNeeded()函数返回 true，说明用户没有配置多线程IO或者系统动态判断当前不需要使用多线程IO，则直接调用 handleClientsWithPendingWrites() 函数完成客户端响应；
- (3) 判断 io_threads_active全局变量的值， 如果为0，说明 IO 线程还没有激活，则调用 startThreadedIO() 函数启动 IO 线程；如果不为0， 则无需调用；再看看是不是要打日志；
- (4) 将数组 server.clients_pending_write中存放的待响应客户端按照对server.io_threads_num取余的方式分配到各个IO线程的任务列表io_threads_list[target_id] 中；
- (5) 设置io_threads_op这个全局IO操作标志的初始状态为IO_THREADS_OP_WRITE(表示IO 线程都处理写任务)，更新io_threads_pending数组的值；
- (6) 自旋检测所有ios线程的处理队列对否都已清空，等待 IO 线程处理任务完毕跳出循环. 这部分逻辑主要靠io_threads_pending数组记录每个IO线程待处理的client数量来判断，如果各个 IO 线程待处理的 client 数量相加为 0，则任务处理完毕，主线程跳出循环；
- (7) 如果还有待处理的客户端则继续处理，处理完毕清空 server.clients_pending_write 数组；
- (8) 再次处理客户端的请求，在需要安装写处理函数的地方安装写处理函数；
- (9) 更新服务的stat_io_writes_processed值.
int handleClientsWithPendingWritesUsingThreads(void) {
/* 步骤1 */
int processed = listLength(server.clients_pending_write);
if (processed == 0) return 0; /* Return ASAP if there are no clients. */
/* 步骤2 */
/* If I/O threads are disabled or we have few clients to serve, don't
* use I/O threads, but thejboring synchronous code. */
if (server.io_threads_num == 1 || stopThreadedIOIfNeeded()) {
return handleClientsWithPendingWrites();
}
/* 步骤3 */
/* Start threads if needed. */
if (!server.io_threads_active) startThreadedIO();
if (tio_debug) printf("%d TOTAL WRITE pending clients\n", processed);
/* 步骤4 */
/* Distribute the clients across N different lists. */
listIter li;
listNode *ln;
listRewind(server.clients_pending_write,&li);
int item_id = 0;
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
c->flags &= ~CLIENT_PENDING_WRITE;
int target_id = item_id % server.io_threads_num;
listAddNodeTail(io_threads_list[target_id],c);
item_id++;
}
/* 步骤5 */
/* Give the start condition to the waiting threads, by setting the
* start condition atomic var. */
io_threads_op = IO_THREADS_OP_WRITE;
for (int j = 1; j < server.io_threads_num; j++) {
int count = listLength(io_threads_list[j]);
io_threads_pending[j] = count;
}
/* 步骤6 */
/* Also use the main thread to process a slice of clients. */
listRewind(io_threads_list[0],&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
writeToClient(c,0);
}
listEmpty(io_threads_list[0]);
/* 步骤7 */
/* Wait for all the other threads to end their work. */
while(1) {
unsigned long pending = 0;
for (int j = 1; j < server.io_threads_num; j++)
pending += io_threads_pending[j];
if (pending == 0) break;
}
if (tio_debug) printf("I/O WRITE All threads finshed\n");
/* 步骤8 */
/* Run the list of clients again to install the write handler where
* needed. */
listRewind(server.clients_pending_write,&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
/* Install the write handler if there are pending writes in some
* of the clients. */
if (clientHasPendingReplies(c) &&
connSetWriteHandler(c->conn, sendReplyToClient) == AE_ERR)
{
freeClientAsync(c);
}
}
listEmpty(server.clients_pending_write);
/* 步骤9 */
/* Update processed count on server */
server.stat_io_writes_processed += processed;
return processed;
}3.3.6 稍微看下startThreadedIO()#networking.c
此函数很简单，主要内容是：
(1)将创建IO线程时锁住的互斥对象解锁, 使IO线程得以运行;
(2)将全局变量io_threads_active赋值为1, 标志IO线程已经激活.
void startThreadedIO(void) {
if (tio_debug) { printf("S"); fflush(stdout); }
if (tio_debug) printf("--- STARTING THREADED IO ---\n");
serverAssert(server.io_threads_active == 0);
for (int j = 1; j < server.io_threads_num; j++)
pthread_mutex_unlock(&io_threads_mutex[j]);
server.io_threads_active = 1;
}3.3.7 阅读readQueryFromClient()#networking.c中与IO多线程中的子线程相关部分
IO线程处理网络读取的主要流程在readQueryFromClient()函数中，该函数主要负责解析客户端传输过来的命令
及参数，其和多线程IO相关的部分为调用networking.c#postponeClientRead()函数将客户端放入等待队中.
需注意如果客户端成功入队(即postponeClientRead(c)返回非0), 则 readQueryFromClient()函数不再继续
执行,直接return.
看看readQueryFromClient
void readQueryFromClient(connection *conn) {
client *c = connGetPrivateData(conn);
int nread, readlen;
size_t qblen;
/* Check if we want to read from the client later when exiting from
* the event loop. This is the case if threaded I/O is enabled. */
if (postponeClientRead(c)) return;
...
}
接下来我们看看postponeClientRead函数：
3.3.8 对于IO多线程中的子线程有意义的postponeClientRead#networking.c
postponeClientRead函数需要判断多种条件才能决定客户端是否能入队，具体如下：
- io_threads_active,必须为1，即IO线程必须是激活状态；
- server.io_threads_do_reads 用户配置必须是运行使用 IO 线程读取数据；
- ProcessingEventsWhileBlocked的值不能为0；
- 客户端的标志位flags必须不是CLIENT_MASTER、CLIENT_SLAVE、CLIENT_PENDING_READ；
以上条件都满足，则将客户端的 flags 标志位设置为CLIENT_PENDING_READ，并将其入队到server.clients_pending_read数组，并返回1；否则返回0.
/* Return 1 if we want to handle the client read later using threaded I/O.
* This is called by the readable handler of the event loop.
* As a side effect of calling this function the client is put in the
* pending read clients and flagged as such. */
int postponeClientRead(client *c) {
if (server.io_threads_active &&
server.io_threads_do_reads &&
!ProcessingEventsWhileBlocked &&
!(c->flags & (CLIENT_MASTER|CLIENT_SLAVE|CLIENT_PENDING_READ)))
{
c->flags |= CLIENT_PENDING_READ;
listAddNodeHead(server.clients_pending_read,c);
return 1;
} else {
return 0;
}
}3.3.9 阅读readQueryFromClient()#networking.c中与IO多线程中的主线程相关部分
如3.3.7和3.3.8中所说，如果没能将消息将入到server.clients_pending_read队列中,
就会在主线程中执行postponeClientRead函数下面的内容：
void readQueryFromClient(connection *conn) {
client *c = connGetPrivateData(conn);
int nread, readlen;
size_t qblen;
/* Check if we want to read from the client later when exiting from
* the event loop. This is the case if threaded I/O is enabled. */
if (postponeClientRead(c)) return;
/* Update total number of reads on server */
server.stat_total_reads_processed++;
readlen = PROTO_IOBUF_LEN;
/* If this is a multi bulk request, and we are processing a bulk reply
* that is large enough, try to maximize the probability that the query
* buffer contains exactly the SDS string representing the object, even
* at the risk of requiring more read(2) calls. This way the function
* processMultiBulkBuffer() can avoid copying buffers to create the
* Redis Object representing the argument. */
if (c->reqtype == PROTO_REQ_MULTIBULK && c->multibulklen && c->bulklen != -1
&& c->bulklen >= PROTO_MBULK_BIG_ARG)
{
ssize_t remaining = (size_t)(c->bulklen+2)-sdslen(c->querybuf);
/* Note that the 'remaining' variable may be zero in some edge case,
* for example once we resume a blocked client after CLIENT PAUSE. */
if (remaining > 0 && remaining < readlen) readlen = remaining;
}
qblen = sdslen(c->querybuf);
if (c->querybuf_peak < qblen) c->querybuf_peak = qblen;
c->querybuf = sdsMakeRoomFor(c->querybuf, readlen);
nread = connRead(c->conn, c->querybuf+qblen, readlen);
if (nread == -1) {
if (connGetState(conn) == CONN_STATE_CONNECTED) {
return;
} else {
serverLog(LL_VERBOSE, "Reading from client: %s",connGetLastError(c->conn));
freeClientAsync(c);
return;
}
} else if (nread == 0) {
serverLog(LL_VERBOSE, "Client closed connection");
freeClientAsync(c);
return;
} else if (c->flags & CLIENT_MASTER) {
/* Append the query buffer to the pending (not applied) buffer
* of the master. We'll use this buffer later in order to have a
* copy of the string applied by the last command executed. */
c->pending_querybuf = sdscatlen(c->pending_querybuf,
c->querybuf+qblen,nread);
}
sdsIncrLen(c->querybuf,nread);
c->lastinteraction = server.unixtime;
if (c->flags & CLIENT_MASTER) c->read_reploff += nread;
server.stat_net_input_bytes += nread;
if (sdslen(c->querybuf) > server.client_max_querybuf_len) {
sds ci = catClientInfoString(sdsempty(),c), bytes = sdsempty();
bytes = sdscatrepr(bytes,c->querybuf,64);
serverLog(LL_WARNING,"Closing client that reached max query buffer length: %s (qbuf initial bytes: %s)", ci, bytes);
sdsfree(ci);
sdsfree(bytes);
freeClientAsync(c);
return;
}
/* There is more data in the client input buffer, continue parsing it
* in case to check if there is a full command to execute. */
processInputBuffer(c);
}
3.3.10 关于IO线程组中的子线程的任务分配参数io-threads-do-reads
# Setting io-threads to 1 will just use the main thread as usually.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
本段节选自配置文件，IO线程组中的子线程的任务默认是写的，要想让它们变成读的，
可以将io-threads-do-reads的属性值设置为yes.
3.4 BIO线程
3.4.1 BIO线程的产生与说明
一共三个BIO线程，这三个线程分别处理：
(1)关闭文件描述符close(2)系统调用;
(2)AOF磁盘同步fsync;
(3)大键bigkey惰性删除.
这里的BIO我也不尝试着理解是什么意思了，它既是background IO，也是blocking IO.
bio.h中定义了这三个后台任务的操作码：
/* Background job opcodes */
#define BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */
#define BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */
#define BIO_LAZY_FREE 2 /* Deferred objects freeing. */
#define BIO_NUM_OPS 3
BIO的三个初始化线程的实现：
/* Initialize the background system, spawning the thread. */
void bioInit(void) {
pthread_attr_t attr;
pthread_t thread;
size_t stacksize;
int j;
/* Initialization of state vars and objects */
for (j = 0; j < BIO_NUM_OPS; j++) {
pthread_mutex_init(&bio_mutex[j],NULL);
pthread_cond_init(&bio_newjob_cond[j],NULL);
pthread_cond_init(&bio_step_cond[j],NULL);
bio_jobs[j] = listCreate();
bio_pending[j] = 0;
}
/* Set the stack size as by default it may be small in some system */
pthread_attr_init(&attr);
pthread_attr_getstacksize(&attr,&stacksize);
if (!stacksize) stacksize = 1; /* The world is full of Solaris Fixes */
while (stacksize < REDIS_THREAD_STACK_SIZE) stacksize *= 2;
pthread_attr_setstacksize(&attr, stacksize);
/* Ready to spawn our threads. We use the single argument the thread
* function accepts in order to pass the job ID the thread is
* responsible of. */
for (j = 0; j < BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
/* 在这里产生线程 */
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
serverLog(LL_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
bio_threads[j] = thread;
}
}
我们可以看到这三个线程公用一个函数bioProcessBackgroundJobs，下面我们来看看它长什么样子吧.3.4.2 bioProcessBackgroundJobs
- 定义变量job，定义变量type并赋初始值，定义变量sigset，判断type的值是否在3以内，如果不在的话则返回NULL；
- 判断type的值并此来给线程命名，如果是1的话，就命名为bio_close_file，如果是2的话，就命名为bio_aof_fsync，如果是3的话，就命名为bio_lazy_free；
- 设置CPU亲缘性；
- 设置本线程对Cancel信号的反应为【收到信号后设为CANCLED状态】，设置本线程取消动作的执行时机为【立即执行取消动作(退出，要在pthread_setcancelstate的第一个参数是PTHREAD_CANCEL_ENABLE状态生效)】；
- 对当前加锁；
- 首先清空信号集，然后在信号集中加入SIGALRM，再屏蔽当前线程中的SIGALRM信号；
- 进入循环；
- 循环中逻辑：如果listLength(bio_jobs[type]) == 0成立，那么就会阻塞在条件变量上；
- 循环中逻辑：如果listLength(bio_jobs[type]) != 0，那么就从bio_jobs[type]队列中去消息出来赋值给job，释放线程的锁，然后根据type的值去选取不同的逻辑去执行相应的函数：如果是type是1，执行close((long)job->arg1)；如果type是2，则执行【redis_fsync((long)job->arg1)】；如果type是3，则根据job->arg1不为0，则调用lazyfreeFreeObjectFromBioThread(job->arg1)，job->arg2且job->arg3都不为0，则调用lazyfreeFreeDatabaseFromBioThread(job->arg2,job->arg3)，如果job->arg3不为0，则调用lazyfreeFreeSlotsMapFromBioThread(job->arg3) ；其他情况则调用serverPanic；
- 循环中逻辑：如果listLength(bio_jobs[type]) != 0，且执行了9，释放job内存，线程再加锁，删除节点，执行bio_pending[type]--；
- 循环中逻辑：广播唤醒因为对应类型的bio_step_cond被阻塞的线程.
void *bioProcessBackgroundJobs(void *arg) {
/* STEP 1 */
struct bio_job *job;
unsigned long type = (unsigned long) arg;
sigset_t sigset;
/* Check that the type is within the right interval. */
if (type >= BIO_NUM_OPS) {
serverLog(LL_WARNING,
"Warning: bio thread started with wrong type %lu",type);
return NULL;
}
/* STEP 2 */
switch (type) {
case BIO_CLOSE_FILE:
redis_set_thread_title("bio_close_file");
break;
case BIO_AOF_FSYNC:
redis_set_thread_title("bio_aof_fsync");
break;
case BIO_LAZY_FREE:
redis_set_thread_title("bio_lazy_free");
break;
}
/* STEP 3 */
redisSetCpuAffinity(server.bio_cpulist);
/* STEP 4 */
/* Make the thread killable at any time, so that bioKillThreads()
* can work reliably. */
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
/* STEP 5 */
pthread_mutex_lock(&bio_mutex[type]);
/* Block SIGALRM so we are sure that only the main thread will
* receive the watchdog signal. */
/* STEP 6 */
sigemptyset(&sigset);
sigaddset(&sigset, SIGALRM);
if (pthread_sigmask(SIG_BLOCK, &sigset, NULL))
serverLog(LL_WARNING,
"Warning: can't mask SIGALRM in bio.c thread: %s", strerror(errno));
/* STEP 7 */
while(1) {
/* STEP 8 */
listNode *ln;
/* The loop always starts with the lock hold. */
if (listLength(bio_jobs[type]) == 0) {
pthread_cond_wait(&bio_newjob_cond[type],&bio_mutex[type]);
continue;
}
/* STEP 9 */
/* Pop the job from the queue. */
ln = listFirst(bio_jobs[type]);
job = ln->value;
/* It is now possible to unlock the background system as we know have
* a stand alone job structure to process.*/
pthread_mutex_unlock(&bio_mutex[type]);
/* Process the job accordingly to its type. */
if (type == BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == BIO_AOF_FSYNC) {
redis_fsync((long)job->arg1);
} else if (type == BIO_LAZY_FREE) {
/* What we free changes depending on what arguments are set:
* arg1 -> free the object at pointer.
* arg2 & arg3 -> free two dictionaries (a Redis DB).
* only arg3 -> free the skiplist. */
if (job->arg1)
lazyfreeFreeObjectFromBioThread(job->arg1);
else if (job->arg2 && job->arg3)
lazyfreeFreeDatabaseFromBioThread(job->arg2,job->arg3);
else if (job->arg3)
lazyfreeFreeSlotsMapFromBioThread(job->arg3);
} else {
serverPanic("Wrong job type in bioProcessBackgroundJobs().");
}
/* STEP 10 */
zfree(job);
/* Lock again before reiterating the loop, if there are no longer
* jobs to process we'll block again in pthread_cond_wait(). */
pthread_mutex_lock(&bio_mutex[type]);
listDelNode(bio_jobs[type],ln);
bio_pending[type]--;
/* STEP 11 */
/* Unblock threads blocked on bioWaitStepOfType() if any. */
pthread_cond_broadcast(&bio_step_cond[type]);
}
}
3.5 jemalloc后台线程
3.5.1 jemalloc后台线程的产生
(1)main
(2)InitServerLast()
(3)set_jemalloc_bg_thread(server.jemalloc_bg_thread);
(4)je_mallctl("background_thread", NULL, 0, &val, 1);
(5)unlikely(malloc_init())// jemalloc.c:3061
(6)malloc_init_hard
(7)background_thread_create
(8)background_thread_entry
(9)background_work
(10)background_thread0_work
(11)check_background_thread_creation
(12)background_thread_create_signals_masked
(13)pthread_create_wrapper
(14)pthread_create_wrapper中调用了pthread_create_fptr(调用这个函数，这个函数在pthread_create_fptr_init中被定义成dlsym(RTLD_NEXT, "pthread_create");)
3.5.2 jemalloc后台线程的其他一些说明
扫一扫
6665
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
