阿里云备份pms每月初全量与每月其它日期的增量数据脚本

前言

及时保障业务数据，便于及时恢复作业，需要对pms数据进行备份处理

注意

1.备份脚本放置在/home/ops/目录下，方便ops用户也能访问到，并能够及时调试程序

2.python脚本务必不能出现rm -rf的命令，以免造成重大的灾难事故。

备份脚本

#!/usr/bin/evn python
# -*- coding: UTF-8 -*-

import os
import shutil
import time
import subprocess

class BackupData():
    def __init__(self, currentDate=time.strftime("%Y%m%d"), aliEcsUser="ops", sourceBackupCjpmsPath="/mnt/pms/app/zentao/tmp/backup/", destCjpmsBackupPath="/home/ops/"):
        self.currentDate = currentDate
        self.aliEcsUser = aliEcsUser
        self.sourceBackupCjpmsPath = sourceBackupCjpmsPath
        self.destCjpmsBackupPath = destCjpmsBackupPath

    def excuteCommand(self, command):
        subp = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        if subp.poll() != 0:
            return subp.stdout.read().decode("gbk").encode("utf-8")
        else:
            return "failure"

    def createDir(self, createDirPath):
        if os.path.isdir(createDirPath):
            pass
        else:
            os.makedirs(createDirPath)        

    def getDirFile(self, command1, command2):
        excuteCommand = "ls {0} | grep {1}".format(command1, command2)
        getCleanData = BackupData().excuteCommand(excuteCommand).strip("\n").split("\n")
        return getCleanData

    def removeDirFile(self):        
        createDirPath = self.sourceBackupCjpmsPath + self.currentDate
        BackupData().createDir(createDirPath)
        command1 = self.sourceBackupCjpmsPath
        command2 = self.currentDate
        currentData = BackupData().getDirFile(command1, command2)
        for removeData in currentData:
            if removeData == self.currentDate:
                exit
            else:
                removeDir = "mv {0}/{1} {0}/{2}".format(self.sourceBackupCjpmsPath, removeData, self.currentDate)
                BackupData().excuteCommand(removeDir)

    def getAllFilePath(self, backupDate):
        BackupData().removeDirFile()
        allFilePath = []
        for root, dirs, files in os.walk(self.sourceBackupCjpmsPath + backupDate):
            for file in files:
                allFilePath.append(os.path.join(root, file))
        return allFilePath

    def getFileList(self, backupDate):
        fileData = BackupData().getAllFilePath(backupDate)
        fileList = []
        for commonDirPathFile in fileData:
            if ("sql" in commonDirPathFile):
                continue
            else:
                appendFile = "/".join(commonDirPathFile.split("/")[9::])
                if appendFile != "":
                    fileList.append(appendFile)
        return fileList    

    def getDifferentData(self):
        newData = BackupData().getFileList(self.currentDate)
        oldDate = int(self.currentDate) - 1
        for i in range(0, 30):
            oldDateFindData = oldDate - i
            if BackupData().getDirFile(self.sourceBackupCjpmsPath + str(oldDateFindData), str(oldDateFindData)) == ['']:
                continue
            else:
                oldDate = oldDateFindData
                break
        oldData = BackupData().getFileList(str(oldDate))
        newFileData = set(newData) - set(oldData)
        return list(newFileData)

    def cleanTime(self):
        command1 = self.sourceBackupCjpmsPath + self.currentDate
        command2 = self.currentDate       
        if BackupData().getDirFile(command1, command2) == ['']:
            exit

    def backupNewData(self):
        if "01" in self.currentDate:
            exit
        BackupData().cleanTime()
        getNeedBackupData = BackupData().getDifferentData()
        if getNeedBackupData == ['']:
            exit
        command1 = self.sourceBackupCjpmsPath + self.currentDate
        command2 = "file"
        getSourceMainDir = BackupData().getDirFile(command1, command2)[0]
        sourceMainDir = self.sourceBackupCjpmsPath + self.currentDate + '/' + getSourceMainDir
        for backupFile in getNeedBackupData:
            sourceMainFilePath = sourceMainDir + '/' + backupFile
            getBackupFilePath = sourceMainFilePath.replace("/".join(sourceMainDir.split("/")[:7]), self.destCjpmsBackupPath)
            createDestBackDir = "/".join(getBackupFilePath.split("/")[:-1])
            BackupData().createDir(createDestBackDir)
            shutil.copy(sourceMainFilePath, getBackupFilePath)
            if os.path.isdir(createDestBackDir):
                pass
            else:
                os.makedirs(createDestBackDir)
            shutil.copy(sourceMainFilePath, getBackupFilePath)
        command2 = "sql"
        getSqlFileName = BackupData().getDirFile(command1, command2)[0]
        if getSqlFileName == '':
            exit
        createBackupSqlDir = self.destCjpmsBackupPath + self.currentDate
        BackupData().createDir(createBackupSqlDir)
        shutil.copy(self.sourceBackupCjpmsPath + self.currentDate + '/' + getSqlFileName, createBackupSqlDir + '/' + getSqlFileName) 
        chownOpsGrant = "chown -R {0}:{0} {1}*".format(self.aliEcsUser, self.destCjpmsBackupPath)
        BackupData().excuteCommand(chownOpsGrant)

if __name__ == "__main__":
    getCurrentData = BackupData()
    print(getCurrentData.backupNewData())

定时执行脚本任务

• 备份脚本要与pms备份时间错开，pms备份在先

[root@pms ops]# cat >> /etc/crontab <<-EOF
30 2 * * * root /usr/bin/python /home/ops/backupPmscleanData.py &> /dev/null	
EOF

清理备份脚本生成的备份数据

• 因涉及到rm -rf命令操作，请慎用root用户，执行定时任务，一定要小心root用户下使用rm -rf命令

[root@pms ops]# cat >> /etc/crontab <<-EOF
45 22 * * * root cd /home/ops/ && rm -rf $(date "+%Y%m")* &> /dev/null	
EOF

清理pms程序生成的数据，保留近10天的历史数据(亦可保留7天的历史数据)

• 因涉及到rm -rf命令操作，请慎用root用户，执行定时任务，一定要小心root用户下使用rm -rf命令

[root@pms ops]# cat >> /etc/crontab <<-EOF
30 23 * * * root cd /mnt/pms/app/zentao/tmp/backup && ls -t | awk 'NR > 10 {print "rm -rf " $0}' | xargs rm -rf &> /dev/null
EOF

补充

阿里云ecs服务器的pms备份，是每天都进行全量备份，要想直接还原近期的数据，数据未被清理，只不过备份的数据放置到了生成的当天日期的目录下，需要还原，可直接进行还原操作

• 注:
  其*.code、*.file、/或者sql文件直接放在此目录下/mnt/pms/app/zentao/tmp/backup(此为默认备份路径)

结语

… …