Shell命令数据迁移实践

中奖纪录侧

数据量：千万级

实施方案：数据库层面迁移，新数据一一匹配

实施脚本：

1、sql整理出需要导出的数据：

 

select a,b,c,d,e,f,g
 from table_1 
                into outfile '/data/aaa/haha.csv'
fields terminated by ',' optionally enclosed by '\''
lines terminated by '\r\n';
 

2、使用shell分割记录结果，分批导入

 

 

split -a 3 -l 20000 haha.csv target/table2_
  
for i in `ls -1 target `
do
    mysql -s -N -p'xxxxxxx' DPEmidas -e "load data infile '/data/aaa/target/$i'
IGNORE into table table2
fields terminated by ','  optionally enclosed by '\''
lines terminated by '\r\n'"
    mv /data/aaa/target/$i /data/aaa/complete/
    sleep 1
done

 

3、

获取 所有项目的git地址：
git/config文件如下：
[core]
	repositoryformatversion = 0
	filemode = false
	bare = false
	logallrefupdates = true
	symlinks = false
	ignorecase = true
[remote "origin"]
	url = ssh://git@git.dianpingoa.com/emidas/emidas-decorate.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[branch "feature/search"]
	remote = origin
	merge = refs/heads/feature/search
[branch "beta"]
	remote = origin
	merge = refs/heads/beta


需要获取每一个url 那一行后面的地址，正则需要匹配  空格符  url = xxxx
具体正则 (^\s*)url = (.*)，sed命令中需要转义  \ $ * + ( )

sed -n 's/\(^\s*\)url = \(.*\)/\2/p' */.git/config > test.txt

https://blog.csdn.net/z69183787/article/details/53693739