2020 6.824 的 Raft Lab 3B

前言

做2020的MIT6.824，完成了实验Raft Lab3B，通过了测试，对于之前的Raft实现的实验请参考Raft Lab 2A， Raft Lab 2B 和 Raft Lab 2C 以及Raft3A

Lab3B主要需要完成日志压缩的需求，以保证实用性以及性能上的需求。总的来说，这个实验需要改进的地方很多，尤其是我的代码其实之前一直上从index=0开始的，到这个实验卡壳才发现index=1，于是又修改了之前的Raft代码，也参考了网上大家的已有实现，然后才逐渐实现日志压缩的需求的。

---

一、Overview

1.1 流程

1. KVServer发现log size大于设定好的阈值，通知对应的Raft server discard log，并把log的snapshot传过去
2. Raft server收到KVServer的通知，截断snapshot之前的log，并通知persister保存KVServer传过来的snapshot
3. leader在发送心跳的时候如果发现有新的snapshot persist了，通知followers InstallSnapshot
4. follower 收到InstallSnapshot，与本地log进行对比，跟新log，并通知persister保存leader传过来的snapshot
5. follower通知对应的KVserver，reset kvStore保持其一致性

---

二、Implementation details

2.0.0 新的properties

这个paper Figure13 其实也有讲到，但是本实验并不要求offset，所以就剩下如下的新的属性了，这些属性会在后面InstallSnapshot具体讲到，其中lastIncludedIndex以及lastIncludedTerm也被我放到了KVServer和Raft的属性中了

Arguments:

term leader’s term
lastIncludedIndex the snapshot replaces all entries up through and including this index
lastIncludedTerm term of lastIncludedIndex
data[] raw bytes of the snapshot chunk, starting at offset

Results:

term currentTerm, for leader to update itself

type KVServer struct {
	...
	
	lastIncludedIndex int
}

type Raft struct {
	...
	//snapshot
	lastIncludedIndex int
	lastIncludedTerm  int
}

2.0.1 Helper functions

由于snapshot的引入，log本身的长度就跟index不一致了，需要加上snapshot的长度，于是引入了一些helper functions

func (rf *Raft) getLog(i int) LogEntry {
	return rf.log[i-1-rf.lastIncludedIndex]
}

func (rf *Raft) getLogLen() int {
	return len(rf.log) + rf.lastIncludedIndex
}

func (rf *Raft) convertedIndex(i int) int {
	return i - 1 - rf.lastIncludedIndex
}

func (rf *Raft) getLastLogIndex() int {
	return rf.getLogLen()
}

func (rf *Raft) getLastLogTerm() int {
	lastLogIndex := rf.getLastLogIndex()
	if lastLogIndex <= rf.lastIncludedIndex {
		return -1
	} else {
		return rf.getLog(lastLogIndex).Term
	}
}

2.1 log size detection

KVServer发现log size大于设定好的阈值，通知对应的Raft server discard log，并把log的snapshot传过去

为流程的第一步，检测log size是否过大需要discard，如果满足条件，生成snapshot via getSnapshot()，并且通知下层Raft discard log via kv.rf.DiscardEarlyEntries(kv.lastIncludedIndex, snapshot)

func (kv *KVServer) snapshotMonitor() {
	for {
		if kv.killed() || kv.maxraftstate == -1 {
			return
		}
		if kv.rf.IsExceedLogSize(kv.maxraftstate) {
			//save state
			kv.mu.Lock()
			snapshot := kv.getSnapshot()
			kv.mu.Unlock()

			//tells Raft that it can discard old log entries
			if snapshot != nil {
				kv.rf.DiscardEarlyEntries(kv.lastIncludedIndex, snapshot)
			}
		}
		time.Sleep(1 * time.Millisecond)
	}
}

func (kv *KVServer) getSnapshot() []byte {
	w := new(bytes.Buffer)
	e := labgob.NewEncoder(w)
	e.Encode(kv.kvStore)
	e.Encode(kv.sequenceMapper)

	return w.Bytes()
}

2.2 DiscardEarlyEntries

Raft server收到KVServer的通知，截断snapshot之前的log，并通知persister保存KVServer传过来的snapshot

为流程第二步，Raft discard KVServer传过来index之前的log

1. 跟新lastIncludedTerm
2. 跟新lastIncludedIndex
3. 跟新log
4. persist

func (rf *Raft) DiscardEarlyEntries(index int, snapshot []byte) {
	rf.mu.Lock()
	defer rf.mu.Unlock()
	
	compactLogLen := index - rf.lastIncludedIndex
	if index <= rf.lastIncludedIndex {
		return
	}
	rf.lastIncludedTerm = rf.getLog(index).Term
	rf.lastIncludedIndex = index
	rf.log = append(make([]LogEntry, 0), rf.log[compactLogLen:]...)

	rf.persistRaftStateAndSnapshot(snapshot)
}

下面是一些helper functions作为log persist

func (rf *Raft) persistRaftState() {
	// Your code here (2C).
	data := rf.encodeRaftState()
	rf.persister.SaveRaftState(data)
}

func (rf *Raft) persistRaftStateAndSnapshot(snapshot []byte) {
	data := rf.encodeRaftState()
	rf.persister.SaveStateAndSnapshot(data, snapshot)
}

func (rf *Raft) encodeRaftState() []byte {
	w := new(bytes.Buffer)
	e := labgob.NewEncoder(w)
	e.Encode(rf.currentTerm)
	e.Encode(rf.votedFor)
	e.Encode(rf.log)
	e.Encode(rf.lastIncludedIndex)
	e.Encode(rf.lastIncludedTerm)
	data := w.Bytes()
	return data
}

2.3 applySnapshot to followers

为流程的第三步，leader 通知 follwer InstallSnapshot

leader在发送心跳的时候如果发现有新的snapshot persist了，通知followers InstallSnapshot

func (rf *Raft) kickoffLogAppending() {
	for i := 0; i < len(rf.peers); i++ {
		if i == rf.me {
			continue
		}
		if rf.nextIndex[i] <= rf.lastIncludedIndex {
			rf.applySnapshotTo(i)
		} else {
			rf.applyEntriesTo(i)
		}
	}
}

func (rf *Raft) applySnapshotTo(i int) {
	DPrintf("[%v] applySnapshotTo %v", rf.me, i)
	args := InstallSnapshotArgs{
		Term:              rf.currentTerm,
		LeaderId:          rf.me,
		LastIncludedIndex: rf.lastIncludedIndex,
		LastIncludedTerm:  rf.lastIncludedTerm,
		Data:              rf.persister.ReadSnapshot(),
	}
	reply := InstallSnapshotReply{}

	go func() {
		ok := rf.sendInstallSnapshot(i, &args, &reply)
		if !ok {
			return
		}

		rf.mu.Lock()
		defer rf.mu.Unlock()

		if rf.currentTerm != args.Term {
			return
		}
		
		if reply.Term > rf.currentTerm {
			return
		}

		rf.nextIndex[i] = args.LastIncludedIndex + 1
		rf.matchIndex[i] = args.LastIncludedIndex
		rf.advanceCommitIndex()
	}()
}

2.4 InstallSnapshot

为流程的第四步，follower install snapshot

follower 收到InstallSnapshot，与本地log进行对比，跟新log，并通知persister保存leader传过来的snapshot

installsnap负责跟新follower的log，前面做一些条件检测，保证leader有更大的Term以及更大或者至少相等的lastIncludedIndex，之后就开始跟新follower的log了，两个rules

1. 如果follower有比args.LastIncludedIndex 更长的index，如果Term一直的情况下，删除args.LastIncludedIndex后的log，保留follower后面的log
2. 其它情况，直接清空follower的log即可

这里有个非常需要注意的一点，就是在准备跟KVServer通信的时候 ，需要再次确认rf.lastApplied <= rf.lastIncludedIndex; 没有这个的话测试时有概率出现一致性问题的，这个问题debug了很久后来也是网上发现solution的

func (rf *Raft) InstallSnapshot(args *InstallSnapshotArgs, reply *InstallSnapshotReply) {
	rf.mu.Lock()
	defer rf.mu.Unlock()

	reply.Term = rf.currentTerm
	if args.Term < rf.currentTerm {
		return
	}
	if args.Term > rf.currentTerm {
		rf.convertToFollower(args.Term)
	}

	rf.lastReceived = time.Now()

	if args.LastIncludedIndex <= rf.lastIncludedIndex {
		return
	}

	if args.LastIncludedIndex < rf.getLastLogIndex() {
		if rf.getLog(args.LastIncludedIndex).Term == args.LastIncludedTerm {
			rf.log = append(make([]LogEntry, 0), rf.log[args.LastIncludedIndex-rf.lastIncludedIndex:]...)
		} else {
			rf.log = make([]LogEntry, 0)
		}
	} else { 
		rf.log = make([]LogEntry, 0)
	}

	rf.lastIncludedIndex = args.LastIncludedIndex
	rf.lastIncludedTerm = args.LastIncludedTerm

	rf.lastApplied = Max(rf.lastApplied, rf.lastIncludedIndex)
	rf.persistRaftStateAndSnapshot(args.Data)

	msg := ApplyMsg{
		IsSnapshot: true,
		Snapshot:   rf.persister.ReadSnapshot(),
	}
	if rf.lastApplied > rf.lastIncludedIndex {
		return
	}
	rf.applyCh <- msg
}

2.5 inform kvserver

为流程的第五步，与应用层KVServer通信，更新kvStore

follower通知对应的KVserver，reset kvStore保持其一致性

Raft发送message上一步已经体现出来了，rf.applyCh <- msg；那么接收方收到applyCh需要进行处理判读是否是跟新kvStore

这里涉及到applyCh新加的属性

type ApplyMsg struct {
	...
	//log
	IsSnapshot        bool
	Snapshot          []byte
	LastIncludedIndex int
	LastIncludedTerm  int
}

KVServer需要在serverMonitor中进行处理，如果收到的applyCh为snapshot则进行updateMappersFromSnapshot

func (kv *KVServer) serverMonitor() {
	for {
		...
		case msg := <-kv.applyCh:
			if msg.IsSnapshot {
				kv.updateMappersFromSnapshot(msg.Snapshot)
				continue
			}
		...
	}
}

func (kv *KVServer) updateMappersFromSnapshot(snapshot []byte) {
	kv.mu.Lock()
	defer kv.mu.Unlock()
	r := bytes.NewBuffer(snapshot)
	d := labgob.NewDecoder(r)
	var kvStore map[string]string
	var sequenceMapper map[int64]int64
	if d.Decode(&kvStore) == nil && d.Decode(&sequenceMapper) == nil {
		kv.kvStore, kv.sequenceMapper = kvStore, sequenceMapper
	}
}

2.6 跟新Make函数

这里需要跟新KVServer以及Raft的init初始化函数

KVServer: 在初始化的时候需要读取persister的snapshot，并且通过snapshotMonitor监控log的size是否>maxraftstate

func StartKVServer(servers []*labrpc.ClientEnd, me int, persister *raft.Persister, maxraftstate int) *KVServer {
	...
	kv.persister = persister
	snapshot := kv.persister.ReadSnapshot()
	if snapshot != nil && len(snapshot) > 0 {
		kv.updateMappersFromSnapshot(snapshot)
	}

	// You may need initialization code here.

	go kv.serverMonitor()
	go kv.snapshotMonitor()

	return kv
}

Raft: 也是在初始化的时候跟新rf.lastApplied以及rf.commitIndex

func Make(peers []*labrpc.ClientEnd, me int,
	...
	//log
	rf.lastIncludedIndex = 0
	rf.lastIncludedTerm = 0
	rf.lastApplied = Max(rf.lastApplied, rf.lastIncludedIndex)
	rf.commitIndex = Max(rf.commitIndex, rf.lastIncludedIndex)
	return rf
}

这样，整个流程就走完了

---

三、关于性能调优

实验有新能上的要求，这里分享一下我research(看别人)的方法，非常好用

go test -run TestSnapshotSize should take less than 20 seconds of real time.

解法就是在Start调用的时候，发现leader马上发送心跳 rf.kickoffLogAppending() ，而不是被动地等待100ms的心跳时间

func (rf *Raft) Start(command interface{}) (int, int, bool) {
	rf.mu.Lock()
	defer rf.mu.Unlock()

	index := -1
	term := -1
	isLeader := true

	term = rf.currentTerm
	isLeader = rf.currentState == Leader
	if isLeader {
		index = rf.getLogLen() + 1
		entry := LogEntry{
			Command: command,
			Index:   index,
			Term:    term,
		}
		rf.log = append(rf.log, entry)
		rf.persistRaftState()
		rf.kickoffLogAppending()
	}
	return index, term, isLeader
}

当然，由于速度一下子就上去了，会带来log consistent的问题，前面其中一个需要注意的就是要在发送applymsg前再次rf.lastApplied不能大于rf.lastIncludedIndex

if rf.lastApplied > rf.lastIncludedIndex {
		return
	}
rf.applyCh <- msg

---

总结

做整个实验需要需要想清楚整个流程，剩下就是耐心地多加调试了，之前觉得单步调试可以一招打遍天下无敌手，Raft才发现Log分析才是正道。