1.业务需求
服务器出现故障或者断点情况下能够在最快的时间内不影响员工正常办公，系统运行正常，数据不丢失。
2.功能描述
两台服务器互相作为数据备份，一台出现死机或者故障，另一台服务器能够启到数据备份的作用。
（1）服务器主-备切换：当一台服务器死机或者出现故障后能够进行主-备节点自动切换或者管理员后台主动切换节点工作模式。
（2）服务器故障恢复：死机的或者出现故障的服务器正常了，恢复到初始工作模式，即“双主”模式。
3.服务器工作模式说明
（1）双主模式：表示两台服务器都正常运作，两两之间数据互为备份，创建的虚拟机正常分配到这两台服务器上。
（2）主-备模式：表示工作不正常的服务器作为备份机，正常的作为主服务器，创建的虚拟机都将添加到主服务器上，备份机只是启数据备份的作用。
4.具体功能业务实现
4.1添加服务器开机脚本
在目录/etc/rc.local下添加一行sh /opt/ha/hainit.sh
4.2节点迁移流程：
(1)设置节点主-备关系
(2)添加到服务器开机脚本中
(3)死机节点下所有虚拟机迁移到另一个的节点下
(4)更新节点高可用规则数据库
(5)设置死机节点数据库字段标识为不可用状态
4.3节点回迁流程：
（1）检测两节点正常通信
（2）节点先设置为主-备模式
（3）同步数据
（4）回迁的虚拟机全部关机并undefine掉
（5）节点恢复成双主模式
（6）添加到服务器开机开机脚本中
（7）虚拟机回迁，修改虚拟机状态、端口、节点，最后启动
（8）更新节点高可用规则数据库（节点关系恢复到1状态）
（9）设置之前死机的节点数据库标识为可用状态
（10）回迁的虚拟机标识为无效状态
5.节点高可用规则说明
节点高可用规则表（nodes_relation表）
数据库字段含义说明：
node_id_a：节点A类型（指向/vmpool）
node_id_b：节点B类型（指向/backuppool）
relation_ship：节点之间的主-备关系
1表示A、B节点正常，模式为“双主模式”
2表示A节点死机,B节点正常(A设为备，B设为主)
3表示B节点死机,A节点正常(B设为备，A设为主)
分别对应以下三种情况
drbd节点关系设置命令
（1）A节点
umount /backuppool
drbdadm secondary backuppool
drbdadm primary vmpool
mount /dev/drbd0 /vmpool
B节点
umount /vmpool
drbdadm secondary vmpool
drbdadm primary backuppool
mount /dev/drbd1 /backuppool
（2）A节点
umount /vmpool
umount /backuppool
drbdadm secondary vmpool
drbdadm secondary backuppool
B节点
drbdadm primary vmpool
drbdadm primary backuppool
mount /dev/drbd0 /vmpool
mount /dev/drbd1 /backuppool
（3）A节点
drbdadm primary vmpool
drbdadm primary backuppool
mount /dev/drbd0 /vmpool
mount /dev/drbd1 /backuppool
B节点
umount /vmpool
umount /backuppool
drbdadm secondary vmpool
drbdadm secondary backuppool
6.代码实现
（1）节点迁移核心代码：
@Override
publicJSONObject migrateNode(int nodeId, int dieNodeId) {
JSONObjectjsonObject = new JSONObject();
try{
Nodenode = nodeDao.findById(nodeId);
NodedieNode = nodeDao.findById(dieNodeId);
if(null!=node && null !=dieNode){
//获取死机节点下所有的虚拟机
List<Vm>vmsList = vmManager.getVmByNodeId(dieNode.getId());
//根据死机节点的类型,设置节点关系并添加到服务器开机脚本中
JSONObjectnodesJsonObject = setNodesRelationByDieNodeId(String.valueOf(node.getId()),node.getHostname(),String.valueOf(dieNodeId),dieNode.getHostname());
StringnodesResult = nodesJsonObject.getString("result");
log.loger.info("setnodes relation result ="+nodesResult);
if(Constant.SUCCESS.equalsIgnoreCase(nodesResult)){
//生成虚拟机并修改节点id
Stringresult = TaemNodeUtil.migrate_node(nodeId, dieNodeId);
log.loger.info("migratenode result="+result);
if(Constant.SUCCESS.equalsIgnoreCase(result)){
//保存节点迁移的虚拟机
vmBakRecordManager.saveNodeMigrateVmRecord(vmsList,node.getId(),dieNode.getId());
//修改节点高可用规则数据库
nodesRelationManager.updateNodesRelationByDieNodeId(dieNodeId);
//设置该死机节点标识为不可用状态
setNodeState(dieNode.getId(),Constant.NODES_BAD_STATE);
jsonObject.put("result","success");
}else{
jsonObject.put("result","error");
}
}else{
jsonObject.put("result",nodesResult);
}
}else{
jsonObject.put("result","noNode");
}
}catch (Exception e) {
jsonObject.put("result","exception");
}
returnjsonObject;
}
（2）节点回迁核心代码
@Override
publicJSONObject recoveryNode(int nodeRelationId) {
JSONObjectjsonObject = new JSONObject();
try{
NodesRelationBeannodesRelationBean = nodesRelationManager.getNodesRelationById(nodeRelationId);
if(null!=nodesRelationBean){
StringnodeAIp = nodesRelationBean.getNodeAIp();
StringnodeBIp = nodesRelationBean.getNodeBIp();
StringnodesRelation = nodesRelationBean.getNodeRelation();
StringdieNodeId="";//死的节点Id
StringgoodNodeId="";//好的节点Id
if(Constant.NODES_RELATION_TWO.equals(nodesRelation)){//备-主模式，A节点死机
dieNodeId=nodesRelationBean.getNodeAId();
goodNodeId=nodesRelationBean.getNodeBId();
}elseif(Constant.NODES_RELATION_THREE.equals(nodesRelation)){//主-备模式 ，B节点死机
dieNodeId=nodesRelationBean.getNodeBId();
goodNodeId=nodesRelationBean.getNodeAId();
}
log.loger.info("dieNodeId="+dieNodeId+"goodNodeId="+goodNodeId);
//1.回迁准备(检查节点之间是否正常)
booleannodeAState = Util.isOnline(nodeAIp);
booleannodeBState = Util.isOnline(nodeBIp);
log.loger.info("nodeAState="+nodeAState+"nodeBState="+nodeBState);
if(nodeAState&& nodeBState){
//2、节点设置成备-主(2)或主-备(3)
Stringresl = nodesRelationManager.setNodesRelation(nodeAIp, nodeBIp, nodesRelation);
if(Constant.SUCCESS.equalsIgnoreCase(resl)){
//2、同步数据
syncNodesData();
//3、回迁的虚拟机全部关机并undefine掉
vmBakRecordManager.delMoveBakVm(goodNodeId);
Thread.sleep(1000*2);
//4、节点恢复成主-主关系(1)
Stringresult = nodesRelationManager.setNodesRelation(nodeAIp, nodeBIp,Constant.NODES_RELATION_ONE);
if(Constant.SUCCESS.equalsIgnoreCase(result)){
//5、设置节点主-主-关系到服务器开机启动脚本中
nodesRelationManager.setNodesRealtionStartInit(nodeAIp,nodeBIp,Constant.NODES_RELATION_ONE);
//6、虚拟机回迁，修改虚拟机状态、端口、节点Id,并启动
vmBakRecordManager.updateVmState(dieNodeId,goodNodeId);
//7、更新节点高可用规则
nodesRelationManager.updateNodesRelationByType(Constant.NODES_RELATION_ONE);
//8、设置节点为可用状态
setNodeState(Integer.parseInt(dieNodeId),Constant.NODES_GOOD_STATE);
//9、将回迁的虚拟机标志为无效状态(2)
vmBakRecordManager.setMoveBakVmInvalid(goodNodeId);
log.loger.info("recoverynode success!");
jsonObject.put("result","success");
}else{
log.loger.info("recovernodes relation fail!");
jsonObject.put("result","nodesRelationFail");
}
}else{
log.loger.info("setnodes master slaver relation fail!");
jsonObject.put("result","nodesMaSaFail");
}
}else{
log.loger.info("nodesis no online!");
jsonObject.put("result","nodeNoOnline");
}
}else{
log.loger.info("nodesis no add nodesRelation!");
jsonObject.put("result","noNodesRelation");
}
}catch (Exception e) {
log.loger.error("recoveryNodehappen exception!"+e);
jsonObject.put("result","exception");
}
returnjsonObject;
}
（3）设置节点drbd主-备关系实现
/**
* 注:
* 设置节点关系
* 死机节点执行有返回值的函数runCommand将等待很长时间
*/
publicString setNodesRelation(String nodeAIp,String nodeBIp,String nodesRelation){
log.loger.info("nodeAIp="+nodeAIp+"nodeBIp="+nodeBIp+" nodesRelation="+nodesRelation);
try{
if(Constant.NODES_RELATION_ONE.equals(nodesRelation)){//A 主 B 主
log.loger.info("setnodes Relation one begin!");
StringnodeB_umount_vmpool = String.format("ssh %s umount /vmpool",nodeBIp);
StringnodeA_umount_backuppool = String.format("ssh %s umount/backuppool",nodeAIp);
StringnodeA_secon_backuppool = String.format("ssh %s drbdadm secondarybackuppool",nodeAIp);
StringnodeB_primary_backuppool =String.format("ssh %s drbdadm primary backuppool",nodeBIp);
StringnodeB_secon_vmpool = String.format("ssh %s drbdadm secondaryvmpool",nodeBIp);
StringnodeA_primaty_vmpool = String.format("ssh %s drbdadm primaryvmpool",nodeAIp);
StringnodeB_mount_backuppool = String.format("ssh %s mount /dev/drbd1 /backuppool",nodeBIp);
StringnodeA_mount_vmpool = String.format("ssh %s mount /dev/drbd0 /vmpool",nodeAIp);
StringrelOne_1 = ExecLinuxCMD.runCommand(nodeB_umount_vmpool);
StringrelTwo_1 = ExecLinuxCMD.runCommand(nodeA_umount_backuppool);
log.loger.info("nodeB_umount_vmpool="+relOne_1);
log.loger.info("nodeA_umount_backuppool="+relTwo_1);
Thread.sleep(500*1);
StringrelOne_2 =ExecLinuxCMD.runCommand(nodeA_secon_backuppool);
Thread.sleep(500*1);
StringrelTwo_2 =ExecLinuxCMD.runCommand(nodeB_primary_backuppool);
log.loger.info("nodeA_secon_backuppool="+relOne_2);
log.loger.info("nodeB_primary_backuppool="+relTwo_2);
log.loger.info("setbackuppool relation end!");
StringrelOne_3=ExecLinuxCMD.runCommand(nodeB_secon_vmpool);
Thread.sleep(500*1);
StringrelTwo_3 =ExecLinuxCMD.runCommand(nodeA_primaty_vmpool);
log.loger.info("nodeB_secon_vmpool="+relOne_3);
log.loger.info("nodeA_primaty_vmpool="+relTwo_3);
log.loger.info("setvmpool relation end!");
Thread.sleep(500*1);
StringrelOne_4=ExecLinuxCMD.runCommand(nodeB_mount_backuppool);
log.loger.info("nodeB_mount_backuppool="+relOne_4);
log.loger.info("mountbackuppool end!");
StringrelTwo_4=ExecLinuxCMD.runCommand(nodeA_mount_vmpool);
log.loger.info("nodeA_mount_vmpool="+relTwo_4);
log.loger.info("mountvmpool end!");
log.loger.info("setnodes Relation one end!");
}elseif(Constant.NODES_RELATION_TWO.equals(nodesRelation)){//A节点死机(A设为备，B设为主)
log.loger.info("setnodes Relation two begin!");
StringnodeA_umount_vmpool = String.format("ssh %s umount /vmpool",nodeAIp);
StringnodeA_umount_backuppool = String.format("ssh %s umount /backuppool",nodeAIp);
StringnodeA_secon_vmpool = String.format("ssh %s drbdadm secondaryvmpool",nodeAIp);
StringnodeA_secon_backuppool = String.format("ssh %s drbdadm secondarybackuppool",nodeAIp);
StringnodeB_primary_backuppool = String.format("ssh %s drbdadm primarybackuppool",nodeBIp);
StringnodeB_primaty_vmpool = String.format("ssh %s drbdadm primaryvmpool",nodeBIp);
StringnodeB_mount_vmpool = String.format("ssh %s mount /dev/drbd0 /vmpool",nodeBIp);
StringnodeB_mount_vackuppool = String.format("ssh %s mount /dev/drbd1 /backuppool",nodeBIp);
ExecLinuxCMD.execNoResp(nodeA_umount_vmpool);
ExecLinuxCMD.execNoResp(nodeA_umount_backuppool);
Thread.sleep(500*1);
ExecLinuxCMD.execNoResp(nodeA_secon_vmpool);
ExecLinuxCMD.execNoResp(nodeA_secon_backuppool);
Thread.sleep(500*1);
ExecLinuxCMD.runCommand(nodeB_primary_backuppool);
ExecLinuxCMD.runCommand(nodeB_primaty_vmpool);
Thread.sleep(500*1);
ExecLinuxCMD.runCommand(nodeB_mount_vmpool);
ExecLinuxCMD.runCommand(nodeB_mount_vackuppool);
log.loger.info("setnodes Relation two end!");
}elseif(Constant.NODES_RELATION_THREE.equals(nodesRelation)){//B节点死机(A设为主，B设为备)
log.loger.info("setnodes Relation three begin!");
StringnodeB_umount_vmpool = String.format("ssh %s umount /vmpool",nodeBIp);
StringnodeB_umount_backuppool = String.format("ssh %s umount/backuppool",nodeBIp);
StringnodeB_secon_backuppool = String.format("ssh %s drbdadm secondarybackuppool",nodeBIp);
StringnodeB_secon_vmpool = String.format("ssh %s drbdadm secondaryvmpool",nodeBIp);
StringnodeA_primary_vmpool = String.format("ssh %s drbdadm primaryvmpool",nodeAIp);
StringnodeA_primary_backuppool= String.format("ssh %s drbdadm primarybackuppool",nodeAIp);
StringnodeA_mount_vmpool = String.format("ssh %s mount /dev/drbd0 /vmpool",nodeAIp);
StringnodeA_mount_vackuppool = String.format("ssh %s mount /dev/drbd1 /backuppool",nodeAIp);
ExecLinuxCMD.execNoResp(nodeB_umount_vmpool);
ExecLinuxCMD.execNoResp(nodeB_umount_backuppool);
Thread.sleep(500*1);
ExecLinuxCMD.execNoResp(nodeB_secon_backuppool);
ExecLinuxCMD.execNoResp(nodeB_secon_vmpool);
Thread.sleep(500*1);
ExecLinuxCMD.runCommand(nodeA_primary_vmpool);
ExecLinuxCMD.runCommand(nodeA_primary_backuppool);
Thread.sleep(500*1);
ExecLinuxCMD.runCommand(nodeA_mount_vmpool);
ExecLinuxCMD.runCommand(nodeA_mount_vackuppool);
log.loger.info("setnodes Relation three end!");
}
}catch (Exception e) {
return Constant.ERROR;
}
returnConstant.SUCCESS;
}
7.功能界面截图
节点drbd主-备关系以及服务状态查看
（1）A节点的主备关系
（2）A节点的drbd服务状态
（3）B节点的主备关系
（4）B节点的drbd服务状态
8.命令说明
//查看主备关系
cat /proc/drbd
drbd-overview
//查看节点drbd服务状态
systemctl status drbd.service
//节点高可用配置IP路径
/etc/drbd.d
//查看同步进度
cat /proc/drbd
//注意
DRBD设备角色切换
DRBD设备在角色切换之前，需要在主节点执行umount命令卸载磁盘先，然后再把一台主机上的DRBD角色修改为Primary，最后把当前节点的磁盘挂载
872
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
