一 序
Redis除了使用RDB文件持久化数据库外，还提供了AOF持久化功能，与RDB持久化的区别如下：
- RDB持久化：把当前进程数据生成时间点快照（point-in-time snapshot）保存到硬盘的过程，避免数据意外丢失。
- AOF持久化：以独立日志的方式记录每次写命令，重启时在重新执行AOF文件中的命令达到恢复数据的目的。
AOF 持久化功能的实现可以分为命令追加（append）、文件写入、文件同步（sync）三个步骤。
下面分别来看
二 命令追加
2.1 原理
当 AOF 持久化功能处于打开状态时， 服务器在执行完一个写命令之后， 会以协议格式将被执行的写命令追加到服务器状态的 aof_buf 缓冲区的末尾.源码在server.c initServer:
/* Open the AOF file if needed. */
if (server.aof_state == AOF_ON) {//打开AOF
server.aof_fd = open(server.aof_filename,
O_WRONLY|O_APPEND|O_CREAT,0644);
if (server.aof_fd == -1) {
serverLog(LL_WARNING, "Can't open the append-only file: %s",
strerror(errno));
exit(1);
}
}
aof 缓存定义在 源码在server.h
struct redisServer {
....
/* AOF persistence */
int aof_state; /* AOF_(ON|OFF|WAIT_REWRITE) */
int aof_fsync; /* Kind of fsync() policy */
char *aof_filename; /* Name of the AOF file */
int aof_no_fsync_on_rewrite; /* Don't fsync if a rewrite is in prog. */
int aof_rewrite_perc; /* Rewrite AOF if % growth is > M and... */
off_t aof_rewrite_min_size; /* the AOF file is at least N bytes. */
off_t aof_rewrite_base_size; /* AOF size on latest startup or rewrite. */
off_t aof_current_size; /* AOF current size. */
int aof_rewrite_scheduled; /* Rewrite once BGSAVE terminates. */
pid_t aof_child_pid; /* PID if rewriting process */
list *aof_rewrite_buf_blocks; /* Hold changes during an AOF rewrite. */
sds aof_buf; /* AOF buffer, written before entering the event loop */
int aof_fd; /* File descriptor of currently selected AOF file */
int aof_selected_db; /* Currently selected DB in AOF */
time_t aof_flush_postponed_start; /* UNIX time of postponed AOF flush */
time_t aof_last_fsync; /* UNIX time of last fsync() */
time_t aof_rewrite_time_last; /* Time used by last AOF rewrite run. */
time_t aof_rewrite_time_start; /* Current AOF rewrite start time. */
int aof_lastbgrewrite_status; /* C_OK or C_ERR */
unsigned long aof_delayed_fsync; /* delayed AOF fsync() counter */
int aof_rewrite_incremental_fsync;/* fsync incrementally while rewriting? */
int aof_last_write_status; /* C_OK or C_ERR */
int aof_last_write_errno; /* Valid if aof_last_write_status is ERR */
int aof_load_truncated; /* Don't stop on unexpected AOF EOF. */
/* AOF pipes used to communicate between parent and child during rewrite. */
int aof_pipe_write_data_to_child;
int aof_pipe_read_data_from_parent;
int aof_pipe_write_ack_to_parent;
int aof_pipe_read_ack_from_child;
int aof_pipe_write_ack_to_child;
int aof_pipe_read_ack_from_parent;
int aof_stop_sending_diff; /* If true stop sending accumulated diffs
to child process. */
sds aof_child_diff; /* AOF diff accumulator child side. */
....
}其中aof_buf就是写入缓冲区，为啥这样设计呢？由于Redis是单线程响应命令，所以每次写AOF文件都直接追加到硬盘中，那么写入的性能完全取决于硬盘的负载，所以Redis会将命令写入到缓冲区中，然后执行文件同步操作，再将缓冲区内容同步到磁盘中，这样就很好的保持了高性能。
举个例子， 如果客户端向服务器发送以下命令：
redis> SET KEY VALUE
OK
那么服务器在执行这个 SET 命令之后， 会将以下协议内容追加到 aof_buf 缓冲区的末尾：
*3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n
到底怎么实现的呢？
2.2 过程源码
每个命令调用都会调用call()，源码在server.c ：
call方法的flags标识可以设置以下情况的值：
| flags宏定义值 | 描述 |
| CMD_CALL_NONE | 代表不设标识 |
| CMD_CALL_SLOWLOG | 有此标识的时候，会去检查命令的执行速度，以便决策是否加入慢日志中 |
| CMD_CALL_STATS | 统计命令被执行的数量 |
| CMD_CALL_PROPAGATE_AOF | 如果命令会改变值或客户端强制命令传播，则将命令追加到AOF文件 |
| CMD_CALL_PROPAGATE_REPL | 如果命令会改变值或客户端强制命令传播，则将命令传播给服务器的从节点 |
| CMD_CALL_PROPAGATE | PROPAGATE_AOF和PROPAGATE_REPL两个标识的别名 |
| CMD_CALL_FULL | SLOWLOG,STATS,PROPAGATE三个标识的别名 |
如果call函数的flags没有被设置成CMD_CALL_PROPAGATE_AOF或 CMD_CALL_PROPAGATE_REPL，那么AOF命令追加和从节点复制都将永远不会发生；
void call(client *c, int flags) {
// start 记录命令开始执行的时间
long long dirty, start, duration;
int client_old_flags = c->flags;
/* Sent the command to clients in MONITOR mode, only if the commands are
* not generated from reading an AOF. */
//如果可以的话，将命令发送到 MONITOR(only if 这些命令不是从AOF发出的)
if (listLength(server.monitors) &&
!server.loading &&
!(c->cmd->flags & (CMD_SKIP_MONITOR|CMD_ADMIN)))
{
replicationFeedMonitors(c,server.monitors,c->db->id,c->argv,c->argc);
}
/* Initialization: clear the flags that must be set by the command on
* demand, and initialize the array for additional commands propagation. */
//初始化
c->flags &= ~(CLIENT_FORCE_AOF|CLIENT_FORCE_REPL|CLIENT_PREVENT_PROP);
redisOpArrayInit(&server.also_propagate);
/* Call the command. */
//执行命令
// 保留旧 dirty 计数器值
dirty = server.dirty;
// 计算命令开始执行的时间
start = ustime();
// 执行实现函数
c->cmd->proc(c);
// 计算命令执行耗费的时间
duration = ustime()-start;
// 计算命令执行之后的 dirty 值
dirty = server.dirty-dirty;
if (dirty < 0) dirty = 0;
/* When EVAL is called loading the AOF we don't want commands called
* from Lua to go into the slowlog or to populate statistics. */
// 不将从 Lua 中发出的命令放入 SLOWLOG ，也不进行统计
if (server.loading && c->flags & CLIENT_LUA)
flags &= ~(CMD_CALL_SLOWLOG | CMD_CALL_STATS);
/* If the caller is Lua, we want to force the EVAL caller to propagate
* the script if the command flag or client flag are forcing the
* propagation. */
//当执行命令的是lua脚本的时候，如果命令的flags或者客户端的flags是强制传播行为，
//那么我们将强制命令调用者去传播(打开propagate)lua脚本
if (c->flags & CLIENT_LUA && server.lua_caller) {
if (c->flags & CLIENT_FORCE_REPL)
server.lua_caller->flags |= CLIENT_FORCE_REPL;
if (c->flags & CLIENT_FORCE_AOF)
server.lua_caller->flags |= CLIENT_FORCE_AOF;
}
/* Log the command into the Slow log if needed, and populate the
* per-command statistics that we show in INFO commandstats. */
// 如果有需要，将命令放到 SLOWLOG 里面
if (flags & CMD_CALL_SLOWLOG && c->cmd->proc != execCommand) {
char *latency_event = (c->cmd->flags & CMD_FAST) ?
"fast-command" : "command";
latencyAddSampleIfNeeded(latency_event,duration/1000);
slowlogPushEntryIfNeeded(c->argv,c->argc,duration);
}
if (flags & CMD_CALL_STATS) {
c->lastcmd->microseconds += duration;
c->lastcmd->calls++;
}
/* Propagate the command into the AOF and replication link */
// 将命令复制到 AOF 和 slave 节点
if (flags & CMD_CALL_PROPAGATE &&
(c->flags & CLIENT_PREVENT_PROP) != CLIENT_PREVENT_PROP)
{
int propagate_flags = PROPAGATE_NONE;
/* Check if the command operated changes in the data set. If so
* set for replication / AOF propagation. */
//检查命令操作是否改变数据，若是则进行传播向aof追加和复制（主从or集群）
if (dirty) propagate_flags |= (PROPAGATE_AOF|PROPAGATE_REPL);
/* If the client forced AOF / replication of the command, set
* the flags regardless of the command effects on the data set. */
//如果客户端强制命令向aof追加写入/节点复制，则重置flags为能影响数据的命令
// 强制 REPL 传播
if (c->flags & CLIENT_FORCE_REPL) propagate_flags |= PROPAGATE_REPL;
// 强制 AOF 传播
if (c->flags & CLIENT_FORCE_AOF) propagate_flags |= PROPAGATE_AOF;
/* However prevent AOF / replication propagation if the command
* implementatino called preventCommandPropagation() or similar,
* or if we don't have the call() flags to do so. */
if (c->flags & CLIENT_PREVENT_REPL_PROP ||
!(flags & CMD_CALL_PROPAGATE_REPL))
propagate_flags &= ~PROPAGATE_REPL;
if (c->flags & CLIENT_PREVENT_AOF_PROP ||
!(flags & CMD_CALL_PROPAGATE_AOF))
propagate_flags &= ~PROPAGATE_AOF;
/* Call propagate() only if at least one of AOF / replication
* propagation is needed. */
//调用传播方法
if (propagate_flags != PROPAGATE_NONE)
propagate(c->cmd,c->db->id,c->argv,c->argc,propagate_flags);
}
/* Restore the old replication flags, since call() can be executed
* recursively. */
// 将客户端的 FLAG 恢复到命令执行之前
// 因为 call 可能会递归执行
c->flags &= ~(CLIENT_FORCE_AOF|CLIENT_FORCE_REPL|CLIENT_PREVENT_PROP);
c->flags |= client_old_flags &
(CLIENT_FORCE_AOF|CLIENT_FORCE_REPL|CLIENT_PREVENT_PROP);
/* Handle the alsoPropagate() API to handle commands that want to propagate
* multiple separated commands. Note that alsoPropagate() is not affected
* by CLIENT_PREVENT_PROP flag. */
// 传播额外的命令
if (server.also_propagate.numops) {
int j;
redisOp *rop;
if (flags & CMD_CALL_PROPAGATE) {
for (j = 0; j < server.also_propagate.numops; j++) {
rop = &server.also_propagate.ops[j];
int target = rop->target;
/* Whatever the command wish is, we honor the call() flags. */
if (!(flags&CMD_CALL_PROPAGATE_AOF)) target &= ~PROPAGATE_AOF;
if (!(flags&CMD_CALL_PROPAGATE_REPL)) target &= ~PROPAGATE_REPL;
if (target)
propagate(rop->cmd,rop->dbid,rop->argv,rop->argc,target);
}
}
redisOpArrayFree(&server.also_propagate);
}
server.stat_numcommands++;
}看下propagate 传播到AOF和slave。
void propagate(struct redisCommand *cmd, int dbid, robj **argv, int argc,
int flags)
{
// 传播到 AOF
if (server.aof_state != AOF_OFF && flags & PROPAGATE_AOF)
feedAppendOnlyFile(cmd,dbid,argv,argc);
// 传播到 slave
if (flags & PROPAGATE_REPL)
replicationFeedSlaves(server.slaves,dbid,argv,argc);
}本篇先跳过slave，重点看AOF，就是调用了feedAppendOnlyFile。 之前书上只是说调用了feedAppendOnlyFile实现了AOF，怎么调用的没有细讲。这里就明白了。feedAppendOnlyFile源码在aof.c。
feedAppendOnlyFile主要逻辑：创建一个空的简单动态字符串（sds），将当前所有追加命令操作都追加到这个sds中，最终将这个sds追加到server.aof_buf,上面有对于带有过期信息的command将会统一被转换为PEXPIREAT命令来进行追加。还有就是，这个函数在写入键之前，需要显式的写入一个SELECT命令，以正确的将所有键还原到正确的数据库中。
void feedAppendOnlyFile(struct redisCommand *cmd, int dictid, robj **argv, int argc) {
sds buf = sdsempty();//设置一个空的sds
robj *tmpargv[3];
/* The DB this command was targeting is not the same as the last command
* we appended. To issue a SELECT command is needed. */
//使用 SELECT 命令，显式设置数据库，确保之后的命令被设置到正确的数据库
if (dictid != server.aof_selected_db) {
char seldb[64];
snprintf(seldb,sizeof(seldb),"%d",dictid);
// 构造SELECT命令的协议格式
buf = sdscatprintf(buf,"*2\r\n$6\r\nSELECT\r\n$%lu\r\n%s\r\n",
(unsigned long)strlen(seldb),seldb);
//执行AOf时，当前数据库的ID
server.aof_selected_db = dictid;
}
// 将 EXPIRE 、 PEXPIRE 和 EXPIREAT 都转成 PEXPIREAT（没发算相对时间）
if (cmd->proc == expireCommand || cmd->proc == pexpireCommand ||
cmd->proc == expireatCommand) {
/* Translate EXPIRE/PEXPIRE/EXPIREAT into PEXPIREAT */
buf = catAppendOnlyExpireAtCommand(buf,cmd,argv[1],argv[2]);
} else if (cmd->proc == setexCommand || cmd->proc == psetexCommand) {
// 如果是 SETEX/PSETEX 命令，则转换成 SET and PEXPIREAT
/* Translate SETEX/PSETEX to SET and PEXPIREAT */
//构建set对象
tmpargv[0] = createStringObject("SET",3);
tmpargv[1] = argv[1];
tmpargv[2] = argv[3];
//把set命令按照协议格式追加到buf中
buf = catAppendOnlyGenericCommand(buf,3,tmpargv);
decrRefCount(tmpargv[0]);
//将SETEX/PSETEX命令和键对象按协议格式追加到buf中
buf = catAppendOnlyExpireAtCommand(buf,cmd,argv[1],argv[2]);
} else if (cmd->proc == setCommand && argc > 3) {
//set 带有超期时间 转换为 SET and PEXPIREAT
int i;
robj *exarg = NULL, *pxarg = NULL;
/* Translate SET [EX seconds][PX milliseconds] to SET and PEXPIREAT */
//把set命令按照协议格式追加到buf中
buf = catAppendOnlyGenericCommand(buf,3,argv);
for (i = 3; i < argc; i ++) {
if (!strcasecmp(argv[i]->ptr, "ex")) exarg = argv[i+1];//秒
if (!strcasecmp(argv[i]->ptr, "px")) pxarg = argv[i+1];//毫秒
}
serverAssert(!(exarg && pxarg));
if (exarg)//
buf = catAppendOnlyExpireAtCommand(buf,server.expireCommand,argv[1],
exarg);
if (pxarg)//
buf = catAppendOnlyExpireAtCommand(buf,server.pexpireCommand,argv[1],
pxarg);
} else {
/* All the other commands don't need translation or need the
* same translation already operated in the command vector
* for the replication itself. */
// 其他命令直接按协议格式转换，然后追加到buf中
buf = catAppendOnlyGenericCommand(buf,argc,argv);
}
/* Append to the AOF buffer. This will be flushed on disk just before
* of re-entering the event loop, so before the client will get a
* positive reply about the operation performed.
* 将命令追加到 AOF 缓存中，在重新进入事件循环之前，这些命令会被冲洗到磁盘上，
* 并向客户端返回一个回复。
*/
if (server.aof_state == AOF_ON)
server.aof_buf = sdscatlen(server.aof_buf,buf,sdslen(buf));
/* If a background append only file rewriting is in progress we want to
* accumulate the differences between the child DB and the current one
* in a buffer, so that when the child process will do its work we
* can append the differences to the new append only file.
*/
// 如果后台正在进行重写，那么将命令追加到重写缓存区中，以便我们记录重写的AOF文件于当前数据库的差异
if (server.aof_child_pid != -1)
aofRewriteBufferAppend((unsigned char*)buf,sdslen(buf));
//释放
sdsfree(buf);
}当AOF机制打开时，将命令写入aof_buf。如果此时有rewrite进程在运行，那么还需要将命令写入server.aof_rewrite_buf_blocks（此结构在下面的AOF重写中介绍）
其中这个函数调用了catAppendOnlyGenericCommand 跟catAppendOnlyExpireAtCommand 。catAppendOnlyGenericCommand函数实现了追加命令到缓冲区中，从这个函数中，可以清楚的看到协议是如何生成的。
sds catAppendOnlyGenericCommand(sds dst, int argc, robj **argv) {
char buf[32];
int len, j;
robj *o;
// 格式："*<argc>\r\n"
buf[0] = '*'; //一个command以*开头
len = 1+ll2string(buf+1,sizeof(buf)-1,argc);//写入参数的总个数
buf[len++] = '\r';//紧接着\r
buf[len++] = '\n';//接着是\n
//拼接到dst后面
dst = sdscatlen(dst,buf,len);
// 遍历所有的参数，建立命令的格式：$<command_len>\r\n<command>\r\n
for (j = 0; j < argc; j++) {
o = getDecodedObject(argv[j]);//解码成字符串对象
buf[0] = '$';;//第一个是$
len = 1+ll2string(buf+1,sizeof(buf)-1,sdslen(o->ptr));
buf[len++] = '\r';
buf[len++] = '\n';
dst = sdscatlen(dst,buf,len);//这个时候就是*n\r\n$len\r\n
// 组合 <command>\r\n
dst = sdscatlen(dst,o->ptr,sdslen(o->ptr));
dst = sdscatlen(dst,"\r\n",2);
decrRefCount(o);
}
return dst;//返回还原后的协议内容
}这个函数只是追加一个普通的键，然而一个过期命令的键，需要全部转换为PEXPIREAT，因为必须将相对时间设置为绝对时间，否则还原数据库时，就无法得知该键是否过期，Redis的catAppendOnlyExpireAtCommand()函数实现了这个功能
/* Create the sds representation of an PEXPIREAT command, using
* 'seconds' as time to live and 'cmd' to understand what command
* we are translating into a PEXPIREAT.
* 创建 PEXPIREAT 命令的 sds 表示，
* cmd 参数用于指定转换的源指令， seconds 为 TTL （剩余生存时间）。
* This command is used in order to translate EXPIRE and PEXPIRE commands
* into PEXPIREAT command so that we retain precision in the append only
* file, and the time is always absolute and not relative.
* 这个函数用于将 EXPIRE 、 PEXPIRE 和 EXPIREAT 转换为 PEXPIREAT
* 从而在保证精确度不变的情况下，将过期时间从相对值转换为绝对值（一个 UNIX 时间戳）。
*/
sds catAppendOnlyExpireAtCommand(sds buf, struct redisCommand *cmd, robj *key, robj *seconds) {
long long when;
robj *argv[3];
/* Make sure we can use strtoll */
//取出过期值，以便使用stroll函数
seconds = getDecodedObject(seconds);
//解析为long
when = strtoll(seconds->ptr,NULL,10);
/* Convert argument into milliseconds for EXPIRE, SETEX, EXPIREAT
* 如果过期值的格式为秒，那么将它转换为毫秒
*/
if (cmd->proc == expireCommand || cmd->proc == setexCommand ||
cmd->proc == expireatCommand)
{
when *= 1000;
}
/* Convert into absolute time for EXPIRE, PEXPIRE, SETEX, PSETEX */
// 如果过期值的格式为相对值，那么将它转换为绝对值
if (cmd->proc == expireCommand || cmd->proc == pexpireCommand ||
cmd->proc == setexCommand || cmd->proc == psetexCommand)
{
when += mstime();
}
decrRefCount(seconds);
// 创建一个 PEXPIREAT 命令对象
argv[0] = createStringObject("PEXPIREAT",9);
argv[1] = key;
argv[2] = createStringObjectFromLongLong(when);
// 将命令还原成协议格式，追加到buf
buf = catAppendOnlyGenericCommand(buf, 3, argv);
decrRefCount(argv[0]);
decrRefCount(argv[2]);
return buf;// 返回buf
}2.3 同步缓冲区到文件
既然缓冲区提供了高性能的保障，那么缓冲区中的数据安全问题如何解决呢？只要数据存在于缓冲区，那么就有丢失的危险。那么，如果控制同步的频率呢？Redis中给出了3中缓冲区同步文件的策略。
| 可配置值 | 说明 |
| AOF_FSYNC_ALWAYS | 命令写入aof_buf后调用系统fsync和操作同步到AOF文件，fsync完成后进程程返回 |
| AOF_FSYNC_EVERYSEC | 命令写入aof_buf后调用系统write操作，write完成后线程返回。fsync同步文件操作由进程每秒调用一次 |
| AOF_FSYNC_NO | 命令写入aof_buf后调用系统write操作，不对AOF文件做fsync同步，同步硬盘由操作由操作系统负责 |
如果用户没有主动为 appendfsync 选项设置值， 那么 appendfsync 选项的默认值为 everysec ， 关于 appendfsync 选项的更多信息， 请参考 Redis 项目附带的示例配置文件 redis.conf 。
tips文件的写入和同步
为了提高文件的写入效率， 在现代操作系统中， 当用户调用
write函数， 将一些数据写入到文件的时候， 操作系统通常会将写入数据暂时保存在一个内存缓冲区里面， 等到缓冲区的空间被填满、或者超过了指定的时限之后， 才真正地将缓冲区中的数据写入到磁盘里面。这种做法虽然提高了效率， 但也为写入数据带来了安全问题， 因为如果计算机发生停机， 那么保存在内存缓冲区里面的写入数据将会丢失。
为此， 系统提供了
fsync和fdatasync两个同步函数， 它们可以强制让操作系统立即将缓冲区中的数据写入到硬盘里面， 从而确保写入数据的安全性。
看下源码的实现。
/* Write the append only file buffer on disk.
*将 AOF 缓存写入到文件中。
* Since we are required to write the AOF before replying to the client,
* and the only way the client socket can get a write is entering when the
* the event loop, we accumulate all the AOF writes in a memory
* buffer and write it on disk using this function just before entering
* the event loop again.
* 因为程序需要在回复客户端之前对 AOF 执行写操作。
* 而客户端能执行写操作的唯一机会就是在事件 loop 中，
* 因此，程序将所有 AOF 写累积到缓存中，
* 并在重新进入事件 loop 之前，将缓存写入到文件中。
* About the 'force' argument:
* 关于 force 参数：
* When the fsync policy is set to 'everysec' we may delay the flush if there
* is still an fsync() going on in the background thread, since for instance
* on Linux write(2) will be blocked by the background fsync anyway.
* When this happens we remember that there is some aof buffer to be
* flushed ASAP, and will try to do that in the serverCron() function.
* 当 fsync 策略为每秒钟保存一次时，如果后台线程仍然有 fsync 在执行，
* 那么我们可能会延迟执行冲洗（flush）操作，
* 因为 Linux 上的 write(2) 会被后台的 fsync 阻塞。
当这种情况发生时，说明需要尽快冲洗 aof 缓存，程序会尝试在 serverCron() 函数中对缓存进行冲洗。
* However if force is set to 1 we'll write regardless of the background
* fsync.
* 不过，如果 force 为 1 的话，那么不管后台是否正在 fsync ,程序都直接进行写入。
*/
#define AOF_WRITE_LOG_ERROR_RATE 30 /* Seconds between errors logging. */
void flushAppendOnlyFile(int force) {
ssize_t nwritten;
int sync_in_progress = 0;
mstime_t latency;
// 如果缓冲区中没有数据，直接返回
if (sdslen(server.aof_buf) == 0) return;
// 同步策略是每秒同步一次
if (server.aof_fsync == AOF_FSYNC_EVERYSEC)
// 是否有 SYNC 正在后台进行？
sync_in_progress = bioPendingJobsOfType(BIO_AOF_FSYNC) != 0;
// 同步策略是每秒同步一次，且不是强制同步的
if (server.aof_fsync == AOF_FSYNC_EVERYSEC && !force) {
/* With this append fsync policy we do background fsyncing.
* 当 fsync 策略为每秒钟一次时， fsync 在后台执行。
* If the fsync is still in progress we can try to delay
* the write for a couple of seconds.
* 如果后台仍在执行 FSYNC ，那么我们可以延迟写操作一两秒
*/
if (sync_in_progress) {// 有 fsync 正在后台进行
// 延迟执行flush操作的开始时间为0，表示之前没有延迟过write
if (server.aof_flush_postponed_start == 0) {
/* No previous write postponing, remember that we are
* postponing the flush and return. */
// 之前没有延迟过write操作，那么将延迟write操作的开始时间保存下来，然后就直接返回
server.aof_flush_postponed_start = server.unixtime;
return;
} else if (server.unixtime - server.aof_flush_postponed_start < 2) {
// 如果之前延迟过write操作，如果没到2秒，直接返回，不执行write
/* We were already waiting for fsync to finish, but for less
* than two seconds this is still ok. Postpone again. */
return;
}
/* Otherwise fall trough, and go write since we can't wait
* over two seconds.
* 如果后台还有 fsync 在执行，并且 write 已经推迟 >= 2 秒
* 那么执行写操作（write 将被阻塞） */
server.aof_delayed_fsync++;
serverLog(LL_NOTICE,"Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.");
}
}
/* We want to perform a single write. This should be guaranteed atomic
* at least if the filesystem we are writing is a real physical one.
* While this will save us against the server being killed I don't think
* there is much to do about the whole server stopping for power problems
* or alike */
// 执行write操作，保证写操作是原子操作（真出问题需要修复）
// 设置延迟检测开始的时间
latencyStartMonitor(latency);
// 将缓冲区的内容写到AOF文件中
nwritten = write(server.aof_fd,server.aof_buf,sdslen(server.aof_buf));
latencyEndMonitor(latency);// 设置延迟的时间 = 当前的时间 - 开始的时间
/* We want to capture different events for delayed writes:
* when the delay happens with a pending fsync, or with a saving child
* active, and when the above two conditions are missing.
* We also use an additional event name to save all samples which is
* useful for graphing / monitoring purposes.
* 捕获不同造成延迟write的事件
*/
// 如果正在后台执行同步fsync
if (sync_in_progress) {
// 将latency和"aof-write-pending-fsync"关联到延迟诊断字典中
latencyAddSampleIfNeeded("aof-write-pending-fsync",latency);
// 如果正在执行AOF或正在执行RDB
} else if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) {
// 将latency和"aof-write-active-child"关联到延迟诊断字典中
latencyAddSampleIfNeeded("aof-write-active-child",latency);
} else {
// 将latency和"aof-write-alone"关联到延迟诊断字典中
latencyAddSampleIfNeeded("aof-write-alone",latency);
}
// 将latency和"aof-write-alone"关联到延迟诊断字典中
latencyAddSampleIfNeeded("aof-write",latency);
/* We performed the write so reset the postponed flush sentinel to zero. */
// 执行了write，所以清零延迟flush的时间
server.aof_flush_postponed_start = 0;
// 如果写入的字节数不等于缓存的字节数，发生异常错误
if (nwritten != (signed)sdslen(server.aof_buf)) {
static time_t last_write_error_log = 0;
int can_log = 0;
/* Limit logging rate to 1 line per AOF_WRITE_LOG_ERROR_RATE seconds. */
// 将日志的记录频率限制在每行 AOF_WRITE_LOG_ERROR_RATE 30 秒
if ((server.unixtime - last_write_error_log) > AOF_WRITE_LOG_ERROR_RATE) {
can_log = 1;
last_write_error_log = server.unixtime;
}
/* Log the AOF write error and record the error code. */
// 如果写入错误，尝试写如到errno到日志
if (nwritten == -1) {//没写进去
if (can_log) {
serverLog(LL_WARNING,"Error writing to the AOF file: %s",
strerror(errno));
server.aof_last_write_errno = errno;
}
} else {
if (can_log) {//写了部分出错
serverLog(LL_WARNING,"Short write while writing to "
"the AOF file: (nwritten=%lld, "
"expected=%lld)",
(long long)nwritten,
(long long)sdslen(server.aof_buf));
}
// 尝试移除新追加的不完整内容（将追加的内容截断，删除了追加的内容，恢复成原来的文件）
if (ftruncate(server.aof_fd, server.aof_current_size) == -1) {
if (can_log) {
serverLog(LL_WARNING, "Could not remove short write "
"from the append-only file. Redis may refuse "
"to load the AOF the next time it starts. "
"ftruncate: %s", strerror(errno));
}
} else {
/* If the ftruncate() succeeded we can set nwritten to
* -1 since there is no longer partial data into the AOF. */
nwritten = -1;
}
server.aof_last_write_errno = ENOSPC;
}
/* Handle the AOF write error. */
// 处理写入 AOF 文件时出现的错误
if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
/* We can't recover when the fsync policy is ALWAYS since the
* reply for the client is already in the output buffers, and we
* have the contract with the user that on acknowledged write data
* is synced on disk. */
//如果是写入的策略为每次写入就同步，无法恢复这种策略的写，因为我们已经告知使用者，已经将写的数据同步到磁盘了，因此直接退出程序
serverLog(LL_WARNING,"Can't recover from AOF write error when the AOF fsync policy is 'always'. Exiting...");
exit(1);
} else {
/* Recover from failed write leaving data into the buffer. However
* set an error to stop accepting writes as long as the error
* condition is not cleared. */
//设置执行write操作的状态
server.aof_last_write_status = C_ERR;
/* Trim the sds buffer if there was a partial write, and there
* was no way to undo it with ftruncate(2). */
// 如果只写入了局部，没有办法用ftruncate()函数去恢复原来的AOF文件
if (nwritten > 0) {
// 只能更新当前的AOF文件的大小
server.aof_current_size += nwritten;
// 删除AOF缓冲区写入的字节数（内存收缩）
sdsrange(server.aof_buf,nwritten,-1);
}
return; /* We'll try again on the next call... */
}
} else { // 执行write写入成功
/* Successful write(2). If AOF was in error state, restore the
* OK state and log the event. */
// 更新最后写入状态为 C_OK
if (server.aof_last_write_status == C_ERR) {
serverLog(LL_WARNING,
"AOF write error looks solved, Redis can write again.");
server.aof_last_write_status = C_OK;
}
}
// 更新写入后的 AOF 文件大小
server.aof_current_size += nwritten;
/* Re-use AOF buffer when it is small enough. The maximum comes from the
* arena size of 4k minus some overhead (but is otherwise arbitrary). */
//如果这个缓存足够小，小于4K，那么重用这个缓存，否则释放AOF缓存
if ((sdslen(server.aof_buf)+sdsavail(server.aof_buf)) < 4000) {
sdsclear(server.aof_buf);//将缓存内容清空，重用
} else {
sdsfree(server.aof_buf); // 释放缓存
server.aof_buf = sdsempty();;//创建一个新缓存
}
/* Don't fsync if no-appendfsync-on-rewrite is set to yes and there are
* children doing I/O in the background.
* 如果 no-appendfsync-on-rewrite 选项为开启状态，
* 并且有 BGSAVE 或者 BGREWRITEAOF 正在进行的话，那么不执行 fsync */
if (server.aof_no_fsync_on_rewrite &&
(server.aof_child_pid != -1 || server.rdb_child_pid != -1))
return;
/* Perform the fsync if needed. */
// 总是执行 fsnyc 同步
if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
/* aof_fsync is defined as fdatasync() for Linux in order to avoid
* flushing metadata. */
// 设置延迟检测开始的时间
latencyStartMonitor(latency);
// Linux下调用fdatasync()函数更高效的执行同步
aof_fsync(server.aof_fd); /* Let's try to get this data on the disk */
// 设置延迟的时间 = 当前的时间 - 开始的时间
latencyEndMonitor(latency);
// 将latency和"aof-fsync-always"关联到延迟诊断字典中
latencyAddSampleIfNeeded("aof-fsync-always",latency);
// 更新最后一次执行 fsnyc 的时间
server.aof_last_fsync = server.unixtime;
} else if ((server.aof_fsync == AOF_FSYNC_EVERYSEC &&
server.unixtime > server.aof_last_fsync)) {
// 每秒执行一次同步，并且距离上次 fsync 已经超过 1 秒
// 如果没有正在执行同步，那么在后台开一个线程执行同步
if (!sync_in_progress) aof_background_fsync(server.aof_fd);
// 更新最近一次执行同步的时间
server.aof_last_fsync = server.unixtime;
}
}everysec策略下 将会使用aof_background_fsync方法来刷新。看代码可以知道其实在异步刷新时间过长两秒以内是不会再进入write方法的。并且在刷新没有完成的情况新写入也不会被刷入。这时候崩溃的话丢失的数据可能会大于1秒.
三 加载load
因为Redis命令总是在一个客户端中执行，因此，为了载入AOF文件，需要创建一个关闭监听套接字的伪客户端。AOF文件的载入和写入是相反的过程，因此比较简单.源码在aof.c
/* Replay the append log file. On success C_OK is returned. On non fatal
* error (the append only file is zero-length) C_ERR is returned. On
* fatal error an error message is logged and the program exists.
*加载aof文件
*/
int loadAppendOnlyFile(char *filename) {
struct client *fakeClient; //伪客户端
FILE *fp = fopen(filename,"r");//打开AOF文件
struct redis_stat sb;
int old_aof_state = server.aof_state;//备份当前AOF状态
long loops = 0;
off_t valid_up_to = 0; /* Offset of the latest well-formed command loaded. */
//能打开，但是空文件的特殊情况（初次就是空）
if (fp && redis_fstat(fileno(fp),&sb) != -1 && sb.st_size == 0) {
server.aof_current_size = 0;
fclose(fp);
return C_ERR;
}
//文件打开失败
if (fp == NULL) {
serverLog(LL_WARNING,"Fatal error: can't open the append log file for reading: %s",strerror(errno));
exit(1);
}
/* Temporarily disable AOF, to prevent EXEC from feeding a MULTI
* to the same file we're about to read. */
// 暂时关闭AOF，防止在执行MULTI时，EXEC命令被传播到AOF文件中
server.aof_state = AOF_OFF;
//创建伪client
fakeClient = createFakeClient();
//设置服务器的状态：正在载入
startLoading(fp);
while(1) {
int argc, j;
unsigned long len;
robj **argv;
char buf[128];
sds argsds;
struct redisCommand *cmd;
/* Serve the clients from time to time */
//间隙性处理client的请求
if (!(loops++ % 1000)) {
// ftello(fp)返回当前文件载入的偏移量
// 设置载入时server的状态信息，更新当前载入的进度
loadingProgress(ftello(fp));
// 在服务器被阻塞的状态下，仍然能处理请求
// 因为当前处于载入状态，当client的请求到来时，总是返回loading的状态错误
processEventsWhileBlocked();
}
//将一行文件内容读入到buf,直到遇到"\r\n"
if (fgets(buf,sizeof(buf),fp) == NULL) {
if (feof(fp))//文件读完，跳出
break;
else
goto readerr;
}
// 检查文件格式 "*<argc>\r\n"
if (buf[0] != '*') goto fmterr;
if (buf[1] == '\0') goto readerr;
//取出命令参数
argc = atoi(buf+1);
//至少要有一个参数(被调用的命令)
if (argc < 1) goto fmterr;
// 分配参数列表空间（举例：set key value）
argv = zmalloc(sizeof(robj*)*argc);
fakeClient->argc = argc;//设置伪client的参数
fakeClient->argv = argv;
//遍历参数列表
// "$<command_len>\r\n<command>\r\n"
for (j = 0; j < argc; j++) {
// 读一行内容到buf中，遇到"\r\n"停止
if (fgets(buf,sizeof(buf),fp) == NULL) {
fakeClient->argc = j; /* Free up to j-1. */
freeFakeClientArgv(fakeClient);
goto readerr;
}
//check 格式
if (buf[0] != '$') goto fmterr;
//读取参数的长度
len = strtol(buf+1,NULL,10);
//读取参数的值：初始化一个长度为len的sds
argsds = sdsnewlen(NULL,len);
// 从文件中读出一个len字节长度，将值保存到argsds中
if (len && fread(argsds,len,1,fp) == 0) {
sdsfree(argsds);
fakeClient->argc = j; /* Free up to j-1. */
freeFakeClientArgv(fakeClient);
goto readerr;
}
// 创建一个字符串对象保存读出的参数argsds
argv[j] = createObject(OBJ_STRING,argsds);
// 读两个字节，跳过"\r\n"
if (fread(buf,2,1,fp) == 0) {
fakeClient->argc = j+1; /* Free up to j. */
freeFakeClientArgv(fakeClient);
goto readerr; /* discard CRLF */
}
}
/* Command lookup */
// 查找命令
cmd = lookupCommand(argv[0]->ptr);
if (!cmd) {
serverLog(LL_WARNING,"Unknown command '%s' reading the append only file", (char*)argv[0]->ptr);
exit(1);
}
/* Run the command in the context of a fake client */
// 查找命令
cmd->proc(fakeClient);
/* The fake client should not have a reply */
// 伪client不应该有回复
serverAssert(fakeClient->bufpos == 0 && listLength(fakeClient->reply) == 0);
/* The fake client should never get blocked */
// 伪client不应该是阻塞的
serverAssert((fakeClient->flags & CLIENT_BLOCKED) == 0);
/* Clean up. Command code may have changed argv/argc so we use the
* argv/argc of the client instead of the local variables. */
// 释放伪client的命令及参数列表
freeFakeClientArgv(fakeClient);
// 更新已载入且命令合法的当前文件的偏移量
if (server.aof_load_truncated) valid_up_to = ftello(fp);
}
/* This point can only be reached when EOF is reached without errors.
* If the client is in the middle of a MULTI/EXEC, log error and quit.
* 如果能执行到这里，说明 AOF 文件的全部内容都可以正确地读取，
* 但是，还要调用uxeof 检查 AOF 是否包含未正确结束的事务 */
if (fakeClient->flags & CLIENT_MULTI) goto uxeof;
// 载入成功
loaded_ok: /* DB loaded, cleanup and return C_OK to the caller. */
fclose(fp);//关闭AOF文件
freeFakeClient(fakeClient);//释放伪client
server.aof_state = old_aof_state; //还原AOF状态
stopLoading(); // 设置载入完成的状态
aofUpdateCurrentSize(); //更新服务器状态中，当前AOF文件的大小
server.aof_rewrite_base_size = server.aof_current_size;// 记录前一次重写时的大小
return C_OK;
// 载入时读错误，如果feof(fp)为真，则直接执行 uxeof
readerr: /* Read error. If feof(fp) is true, fall through to unexpected EOF. */
if (!feof(fp)) {
// 退出前释放伪client的空间
if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
serverLog(LL_WARNING,"Unrecoverable error reading the append only file: %s", strerror(errno));
exit(1);
}
// 读入错误
uxeof: /* Unexpected AOF end of file. */
// 如果发现末尾结束格式不完整则自动截掉,成功加载前面正确的数据。（可能是 AOF 文件在写入的中途遭遇了停机）
if (server.aof_load_truncated) {
serverLog(LL_WARNING,"!!! Warning: short read while loading the AOF file !!!");
serverLog(LL_WARNING,"!!! Truncating the AOF at offset %llu !!!",
(unsigned long long) valid_up_to);
// 截断文件到正确加载的位置
if (valid_up_to == -1 || truncate(filename,valid_up_to) == -1) {
if (valid_up_to == -1) {
serverLog(LL_WARNING,"Last valid command offset is invalid");
} else {
serverLog(LL_WARNING,"Error truncating the AOF file: %s",
strerror(errno));
}
} else {
/* Make sure the AOF file descriptor points to the end of the
* file after the truncate call. */
// 确保截断后的文件指针指向文件的末尾
if (server.aof_fd != -1 && lseek(server.aof_fd,0,SEEK_END) == -1) {
serverLog(LL_WARNING,"Can't seek the end of the AOF file: %s",
strerror(errno));
} else {
serverLog(LL_WARNING,
"AOF loaded anyway because aof-load-truncated is enabled");
goto loaded_ok;//跳转到loaded_ok，表截断成功，成功加载前面正确的数据。
}
}
}
// 退出前释放伪client的空间
if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
serverLog(LL_WARNING,"Unexpected end of file reading the append only file. You can: 1) Make a backup of your AOF file, then use ./redis-check-aof --fix <filename>. 2) Alternatively you can set the 'aof-load-truncated' configuration option to yes and restart the server.");
exit(1);
// 格式错误
fmterr: /* Format error. */
// 退出前释放伪client的空间
if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
serverLog(LL_WARNING,"Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>");
exit(1);
}四 重写机制Rewrite
因为AOF的运作方式是不断的向AOF文件中追加命令，所以随着写入命令的不断增加，AOF文件的体积会变得越来越大.为解决AOF文件过大的问题，Redis引入了AOF文件重写（rewrite）功能，执行BGREWRITEAOF 命令，Redis 将生成一个新的 AOF 文件，这个文件包含重建当前数据集所需的最少命令。
4.1 AOF重写的方式
- 进程内已经超时的数据不在写入文件。
- 无效命令不在写入文件。
- 多条写的命令合并成一个。
总之，AOF总是记录数据库的最终状态的一个命令集。类似于物理中的位移与路程的关系，位移总是关心的是启动到终点距离，而不关心是如何从起点到达终点。
4.2 触发机制
手动触发：BGREWRITEAOF 命令。
自动触发：根据redis.conf的两个参数确定触发的时机。
auto-aof-rewrite-percentage 100：当前AOF的文件空间(aof_current_size)和上一次重写后AOF文件空间(aof_base_size)的比值。
auto-aof-rewrite-min-size 64mb：表示运行AOF重写时文件最小的体积。
自动触发时机 = (aof_current_size > auto-aof-rewrite-min-size && (aof_current_size - aof_base_size) / aof_base_size >= auto-aof-rewrite-percentage)
4.3 重写的实现
AOF 重写程序可以很好地完成创建一个新 AOF 文件的任务，但是，在执行这个程序的时候，调用者线程会被阻塞 。作为一种辅佐性的维护手段，Redis 不希望 AOF 重写造成服务器无法处理请求，所以Redis 决定将 AOF 重写程序放到（后台）子进程里执行，这样处理的最大好处是：
1. 子进程进行 AOF 重写期间，主进程可以继续处理命令请求
2. 子进程带有主进程的数据副本，使用子进程而不是线程，可以在避免锁的情况下，保证数据的安全性不过
使用子进程也有一个问题需要解决：因为子进程在进行 AOF 重写期间，主进程还需要继续处理命令，而新的命令可能对现有的数据进行修改，这会让当前数据库的数据和重写后的AOF 文件中的数据不一致。
为了解决这个问题，Redis 增加了一个 AOF 重写缓冲区，这个缓冲区在 fork 出子进程之后开始启用，Redis 主进程在接到新的写命令之后，将这个写命令的协议内容追加到 AOF缓冲区，AOF重写缓冲区。
这个缓存就是上文提到的server.aof_rewrite_buf_blocks
换言之，当子进程在执行 AOF 重写时，主进程需要执行以下三个工作：
- 1. 处理命令请求
- 2. 将写命令追加到现有的 AOF 缓冲区（server.aof_buf）
- 3. 将写命令追加到 AOF 重写缓存中（server.aof_rewrite_buf_blocks ）
这样一来可以保证：
- 1. 现有的 AOF 功能会继续执行，即使在 AOF 重写期间发生停机，也不会有任何数据丢失
- 2. 所有对数据库进行修改的命令都会被记录到 AOF 重写缓存中
当子进程完成 AOF 重写之后，它会向父进程发送一个完成信号，父进程在接到完成信号之后，会调用一个信号处理函数，并完成以下工作：
- 1. 将 AOF 重写缓存中的内容全部写入到新 AOF 文件中
- 2. 对新的 AOF 文件进行改名，覆盖原有的 AOF 文件
当步骤 1 执行完毕之后，现有 AOF 文件、新 AOF 文件和数据库三者的状态就完全一致
当步骤 2 执行完毕之后，程序就完成了新旧两个 AOF 文件的交替
这个信号处理函数执行完毕之后，主进程就可以继续像往常一样接受命令请求了。在整个 AOF后台重写过程中，只有最后的写入缓存和改名操作会造成主进程阻塞，在其他时候，AOF 后台重写都不会对主进程造成阻塞，这将 AOF 重写对性能造成的影响降到了最低。
4.4 源码：
看下源码的实现，从bgrewriteaofCommand 开始
void bgrewriteaofCommand(client *c) {
if (server.aof_child_pid != -1) { // 不能重复运行 BGREWRITEAOF
addReplyError(c,"Background append only file rewriting already in progress");
} else if (server.rdb_child_pid != -1) {
// 如果正在执行 BGSAVE ，那么预定 BGREWRITEAOF
// 等 BGSAVE 完成之后， BGREWRITEAOF 就会开始执行
server.aof_rewrite_scheduled = 1;
addReplyStatus(c,"Background append only file rewriting scheduled");
} else if (rewriteAppendOnlyFileBackground() == C_OK) {
// 执行 BGREWRITEAOF
addReplyStatus(c,"Background append only file rewriting started");
} else {
addReply(c,shared.err);
}
}服务器主进程函数：rewriteAppendOnlyFileBackground执行了fork操作生成一个子进程执行rewriteAppendOnlyFile()函数进行对临时文件的重写操作。源码如下：
/* This is how rewriting of the append only file in background works:
* 以下是后台重写 AOF 文件（BGREWRITEAOF）的工作步骤：
* 1) The user calls BGREWRITEAOF
* 用户调用 BGREWRITEAOF
* 2) Redis calls this function, that forks():
* Redis 调用这个函数，它执行 fork() ：
* 2a) the child rewrite the append only file in a temp file.
* 子进程在临时文件中对 AOF 文件进行重写
* 2b) the parent accumulates differences in server.aof_rewrite_buf.
父进程将新输入的写命令追加到 server.aof_rewrite_buf 中
* 3) When the child finished '2a' exists.
当步骤 2a 执行完之后，子进程结束
* 4) The parent will trap the exit code, if it's OK, will append the
* data accumulated into server.aof_rewrite_buf into the temp file, and
* finally will rename(2) the temp file in the actual file name.
* The the new file is reopened as the new append only file. Profit!
* 父进程会捕捉子进程的退出信号，
* 如果子进程的退出状态是 OK 的话，
* 那么父进程将新输入命令的缓存追加到临时文件，
* 然后使用 rename(2) 对临时文件改名，用它代替旧的 AOF 文件，
* 至此，后台 AOF 重写完成。
*/
int rewriteAppendOnlyFileBackground(void) {
pid_t childpid;
long long start;
// 如果正在进行重写或正在进行RDB持久化操作，则返回C_ERR
if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR;
// 创建父子进程间通信的管道
if (aofCreatePipes() != C_OK) return C_ERR;
// 记录fork()开始时间（计算 fork 耗时用）
start = ustime();
//子进程
if ((childpid = fork()) == 0) {
char tmpfile[256];
/* Child */
// 关闭监听的套接字
closeListeningSockets(0);
// 设置进程名字
redisSetProcTitle("redis-aof-rewrite");
// 创建临时文件
snprintf(tmpfile,256,"temp-rewriteaof-bg-%d.aof", (int) getpid());
// 对临时文件进行AOF重写
if (rewriteAppendOnlyFile(tmpfile) == C_OK) {
// 获取子进程使用的内存空间大小
size_t private_dirty = zmalloc_get_private_dirty();
if (private_dirty) {
serverLog(LL_NOTICE,
"AOF rewrite: %zu MB of memory used by copy-on-write",
private_dirty/(1024*1024));
} // 发送成功退出子进程信号（具体实现在server.c）
exitFromChild(0);
} else { // 异常退出子进程
exitFromChild(1);
}
} else {
/* Parent 父进程 */
// 记录执行 fork 所消耗的时间
server.stat_fork_time = ustime()-start;
// 计算fork的速率，GB/每秒
server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
// 将"fork"和fork消耗的时间关联到延迟诊断字典中
latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);
if (childpid == -1) {// 如果创建子进程失败，直接返回
serverLog(LL_WARNING,
"Can't rewrite append only file in background: fork: %s",
strerror(errno));
aofClosePipes();
return C_ERR;
}
// 打印日志
serverLog(LL_NOTICE,
"Background append only file rewriting started by pid %d",childpid);
// 记录 AOF 重写的信息
server.aof_rewrite_scheduled = 0; // 将AOF日程标志清零
server.aof_rewrite_time_start = time(NULL); // AOF开始的时间
server.aof_child_pid = childpid;// 设置AOF重写的子进程pid
updateDictResizePolicy();// 在AOF或RDB期间，不能对哈希表进行resize操作
/* We set appendseldb to -1 in order to force the next call to the
* feedAppendOnlyFile() to issue a SELECT command, so the differences
* accumulated by the parent into server.aof_rewrite_buf will start
* with a SELECT statement and it will be safe to merge.
* 将 aof_selected_db 设为 -1 ，
* 强制让 feedAppendOnlyFile() 下次执行时引发一个 SELECT 命令，
* 从而确保之后新添加的命令会设置到正确的数据库中*/
server.aof_selected_db = -1;
// 清空脚本缓存
replicationScriptCacheFlush();
return C_OK;
}
return C_OK; /* unreached */
}rewrite任务就交给子进程运行rewriteAppendOnlyFile()解决，再看：rewriteAppendOnlyFile
/* Write a sequence of commands able to fully rebuild the dataset into
* "filename". Used both by REWRITEAOF and BGREWRITEAOF.
* 将一系列足以还原当前数据集的命令写入到 filename 指定的文件中。
* In order to minimize the number of commands needed in the rewritten
* log Redis uses variadic commands when possible, such as RPUSH, SADD
* and ZADD. However at max AOF_REWRITE_ITEMS_PER_CMD items per time
* are inserted using a single command.
*为了使重建数据集的命令数量最小，Redis会使用 可变参的命令，例如RPUSH, SADD 和 ZADD。
* 然而每次单个命令的元素数量不能超过AOF_REWRITE_ITEMS_PER_CMD
*/
int rewriteAppendOnlyFile(char *filename) {
dictIterator *di = NULL;
dictEntry *de;
rio aof;
FILE *fp;
char tmpfile[256];
int j;
long long now = mstime();
char byte;
size_t processed = 0;
/* Note that we have to use a different temp name here compared to the
* one used by rewriteAppendOnlyFileBackground() function.
* 创建临时文件
* 注意这里创建的文件名和 rewriteAppendOnlyFileBackground() 创建的文件名稍有不同 */
snprintf(tmpfile,256,"temp-rewriteaof-%d.aof", (int) getpid());
// 打开文件
fp = fopen(tmpfile,"w");
if (!fp) {
serverLog(LL_WARNING, "Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): %s", strerror(errno));
return C_ERR;
}
// 设置一个空sds给保存子进程AOF时差异累计数据的sds
server.aof_child_diff = sdsempty();
// 初始化rio为文件io对象
rioInitWithFile(&aof,fp);
// 设置每写入 REDIS_AOF_AUTOSYNC_BYTES 字节
// 就执行一次 FSYNC
// 防止缓存中积累太多命令内容，造成 I/O 阻塞时间过长
if (server.aof_rewrite_incremental_fsync)
// 设置自动同步的字节数限制为AOF_AUTOSYNC_BYTES = 32MB
rioSetAutoSync(&aof,AOF_AUTOSYNC_BYTES);
// 遍历所有的数据库
for (j = 0; j < server.dbnum; j++) {
// 按照格式构建 SELECT 命令内容
char selectcmd[] = "*2\r\n$6\r\nSELECT\r\n";
// 当前数据库指针
redisDb *db = server.db+j;
// 数据库的键值对字典
dict *d = db->dict;
// 如果数据库中键值对为空则跳过当前数据库
if (dictSize(d) == 0) continue;
// 创建一个安全的字典迭代器
di = dictGetSafeIterator(d);
if (!di) {
fclose(fp); // 创建失败返回C_ERR
return C_ERR;
}
/* SELECT the new DB */
// 将SELECT 命令写入AOF文件，确保后面的命令能正确载入到数据库
if (rioWrite(&aof,selectcmd,sizeof(selectcmd)-1) == 0) goto werr
// 将数据库的ID写入AOF文件;
if (rioWriteBulkLongLong(&aof,j) == 0) goto werr;
/* Iterate this DB writing every entry */
// 遍历保存当前数据的键值对的字典，并通过命令将它们的当前状态（值）记录到新 AOF 文件中
while((de = dictNext(di)) != NULL) {
sds keystr;
robj key, *o;
long long expiretime;
// 取出键
keystr = dictGetKey(de);
// 取出值
o = dictGetVal(de);
// 初始化一个在栈中分配的键对象
initStaticStringObject(key,keystr);
// 获取该键值对的过期时间
expiretime = getExpire(db,&key);
/* If this key is already expired skip it */
// 如果当前键已经过期，则跳过该键
if (expiretime != -1 && expiretime < now) continue;
/* Save the key and associated value */
// 根据值的对象类型，将键值对写到AOF文件中
// 值为字符串类型对象
if (o->type == OBJ_STRING) {
/* Emit a SET command */
char cmd[]="*3\r\n$3\r\nSET\r\n";
// 按格式写入SET命令
if (rioWrite(&aof,cmd,sizeof(cmd)-1) == 0) goto werr;
/* Key and value */
// 按格式写入键值对对象
if (rioWriteBulkObject(&aof,&key) == 0) goto werr;
if (rioWriteBulkObject(&aof,o) == 0) goto werr;
// 值为列表类型对象
} else if (o->type == OBJ_LIST) {
if (rewriteListObject(&aof,&key,o) == 0) goto werr;
// 值为集合类型对象
} else if (o->type == OBJ_SET) {
if (rewriteSetObject(&aof,&key,o) == 0) goto werr;
// 值为有序集合类型对象
} else if (o->type == OBJ_ZSET) {
if (rewriteSortedSetObject(&aof,&key,o) == 0) goto werr;
// 值为哈希类型对象
} else if (o->type == OBJ_HASH) {
if (rewriteHashObject(&aof,&key,o) == 0) goto werr;
} else {
serverPanic("Unknown object type");
}
/* Save the expire time */
// 如果该键有过期时间，且没过期，写入过期时间
if (expiretime != -1) {
char cmd[]="*3\r\n$9\r\nPEXPIREAT\r\n";
// 将过期键时间全都以Unix时间写入（ PEXPIREAT expiretime 命令）
if (rioWrite(&aof,cmd,sizeof(cmd)-1) == 0) goto werr;
if (rioWriteBulkObject(&aof,&key) == 0) goto werr;
if (rioWriteBulkLongLong(&aof,expiretime) == 0) goto werr;
}
/* Read some diff from the parent process from time to time. */
// 在rio的缓存中每次写了10M，就从父进程读累计的差异，保存到子进程的aof_child_diff中
if (aof.processed_bytes > processed+1024*10) {
// 更新已写的字节数
processed = aof.processed_bytes;
// 从父进程读累计写入的缓冲区的差异，在重写结束时链接到文件的结尾
aofReadDiffFromParent();
}
}
dictReleaseIterator(di); //释放字典迭代器
di = NULL;
}
/* Do an initial slow fsync here while the parent is still sending
* data, in order to make the next final fsync faster. */
// 当父进程仍然在发送数据时，先执行一个缓慢的同步，以便下一次最中的同步更快
if (fflush(fp) == EOF) goto werr;
if (fsync(fileno(fp)) == -1) goto werr;
/* Read again a few times to get more data from the parent.
* We can't read forever (the server may receive data from clients
* faster than it is able to send data to the child), so we try to read
* some more data in a loop as soon as there is a good chance more data
* will come. If it looks like we are wasting time, we abort (this
* happens after 20 ms without new data). */
// 再次从父进程读取几次数据，以获得更多的数据，我们无法一直读取，因为服务器从client接受的数据总是比发送给子进程要快
// 所以当数据来临的时候，我们尝试从在循环中多次读取。如果在20ms之内没有新的数据到来，那么我们终止读取
int nodata = 0;
mstime_t start = mstime();//读取的开始时间
// 在20ms之内等待数据到来
while(mstime()-start < 1000 && nodata < 20) {
// 在1ms之内，查看从父进程读数据的fd是否变成可读的，若不可读则aeWait()函数返回0
if (aeWait(server.aof_pipe_read_data_from_parent, AE_READABLE, 1) <= 0)
{
nodata++;//更新新数据到来的时间，超过20ms则退出while循环
continue;
}
// 当管道的读端可读时，清零nodata
nodata = 0; /* Start counting from zero, we stop on N *contiguous*
timeouts. */
// 从父进程读累计写入的缓冲区的差异，在重写结束时链接到文件的结尾
aofReadDiffFromParent();
}
/* Ask the master to stop sending diffs. */
// 请求父进程停止发送累计差异数据
if (write(server.aof_pipe_write_ack_to_parent,"!",1) != 1) goto werr;
// 将从父进程读ack的fd设置为非阻塞模式
if (anetNonBlock(NULL,server.aof_pipe_read_ack_from_parent) != ANET_OK)
goto werr;
/* We read the ACK from the server using a 10 seconds timeout. Normally
* it should reply ASAP, but just in case we lose its reply, we are sure
* the child will eventually get terminated. */
// 在5000ms之内，从fd读1个字节的数据保存在byte中，查看byte是否是'!'
if (syncRead(server.aof_pipe_read_ack_from_parent,&byte,1,5000) != 1 ||
byte != '!') goto werr;
// 如果收到的是父进程发来的'!'，则打印日志
serverLog(LL_NOTICE,"Parent agreed to stop sending diffs. Finalizing AOF...");
/* Read the final diff if any. */
// 最后一次从父进程读累计写入的缓冲区的差异
aofReadDiffFromParent();
/* Write the received diff to the file. */
serverLog(LL_NOTICE,
"Concatenating %.2f MB of AOF diff received from parent.",
(double) sdslen(server.aof_child_diff) / (1024*1024));
// 将子进程aof_child_diff中保存的差异数据写到AOF文件中
if (rioWrite(&aof,server.aof_child_diff,sdslen(server.aof_child_diff)) == 0)
goto werr;
/* Make sure data will not remain on the OS's output buffers */
// 再次冲洗文件缓冲区，执行同步操作,并关闭新AOF文件
if (fflush(fp) == EOF) goto werr;
if (fsync(fileno(fp)) == -1) goto werr;
if (fclose(fp) == EOF) goto werr;
/* Use RENAME to make sure the DB file is changed atomically only
* if the generate DB file is ok. */
//原子地改名，用重写后的新 AOF 文件覆盖旧 AOF 文件appendonly.aof
if (rename(tmpfile,filename) == -1) {
serverLog(LL_WARNING,"Error moving temp append only file on the final destination: %s", strerror(errno));
unlink(tmpfile);
return C_ERR;
}
// 打印日志
serverLog(LL_NOTICE,"SYNC append only file rewrite performed");
return C_OK;
werr:// 写错误处理
serverLog(LL_WARNING,"Write error writing append only file on disk: %s", strerror(errno));
fclose(fp);
unlink(tmpfile);
if (di) dictReleaseIterator(di);
return C_ERR;
}
我们可以看到在关闭文件之前，多次执行了从重写缓冲区做读操作的aofReadDiffFromParent()。在最后执行了rioWrite(&aof,server.aof_child_diff,sdslen(server.aof_child_diff)操作，这就是把AOF重写缓冲区保存服务器主进程新命令追加写到AOF文件中，以此保证了AOF文件的数据状态和数据库的状态一致。
总结：
优点 1、数据防丢失 2、数据可恢复 3、文件重写
缺点 1、数据文件庞大 2、恢复数据慢
关于父子进程的通信，这里没有仔细展开，待后面在整理。
参考:
https://blog.csdn.net/men_wen/article/details/71375513
https://blog.csdn.net/qq_28851503/article/details/82289777
154
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
