2021-06-08 gitlab删除大文件

0.背景

开发的时候误将proguardMapping.txt 加入版本管理, 该文件较大,MB级别,每次发包都会生成的文件,导致gitlab 提交Merge Request的时候经常合并失败,提示500

1. 拉取项目:将项目 所有分支 拉下来 (单分支 直接clone即可)

git clone xxx # 默认拉取master分支
cd xx #进入拉取的文件夹
git branch -r | grep -v '\->' | while read remote; do git branch --track "${remote#origin/}" "$remote"; done
git fetch --all
git pull --all

查看删除前查看大小(可不执行,仅做对比)

$ git count-objects -vH
warning: garbage found: .git/objects/80/tmp_obj_RmCfwe
count: 124
size: 30.14 KiB
in-pack: 5507
packs: 1
size-pack: 20.42 MiB
prune-packable: 0
garbage: 1
size-garbage: 172 bytes

2. 查找大文件

git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')"

$ git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')"
21f94c21a02afa2e28dbbcb13640bf77aba59c7f prj/src/main/assets/apu.mp4
cead7aeea3e606ba158e96d02b2ff43eeffda3ca prj/proguardMapping.txt
2b34b6945d81fdc86f72af9e8e58c7b3645b50b2 prj/proguardMapping.txt
9533dfddf0c3dece13e81831c62161bcf9d9f493 prj/proguardMapping.txt
53ee920749181d51f6969b160cc0e38f2ef0d147 prj/proguardMapping.txt
01553b94780b057bbd54d6dfb8025f1f21083541 prj/proguardMapping.txt
b72ff924ecef61b2ba3990b6d60a12f8a519bcc0 prj/proguardMapping.txt
50699882ff36cca85520c372da930c747425b0f6 prj/proguardMapping.txt
c6b4d9d3f07a3da43444dc9f16c2fb694f96a074 prj/proguardMapping.txt
9130775fe603c6834f474ab105307fc1bee94be3 prj/src/main/jniLibs/armeabi-v7a/libopencv_java4.so

3.删除大文件

可以用通配符， 这里用*/proguardMapping.txt

⚠️一次只能删除一个文件或者文件夹

git filter-branch --force --index-filter "git rm -r --cached --ignore-unmatch */proguardMapping.txt" --prune-empty --tag-name-filter cat -- --all

WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...

Rewrite 1ab5a65a4c80d6c196f486925584fd501c382fac (428/510) (206 seconds passed, remaining 39 predicted)    rm 'camerafix/.gitignore'
rm 'camerafix/README.md'
rm 'camerafix/build.gradle'
rm 'camerafix/proguard-rules.pro'

删除每个commit中包含的文件, 出现rm表示该commit包含文件同时删除成功

同时回收本地空间

git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin && git reflog expire --expire=now --all|git gc --prune=now

查看删除后大小

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 4510
packs: 1
size-pack: 12.07 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

4.强制推送到服务器 （分支需要取消保护）

git push origin --force "refs/heads/*" --tags

5.清除服务缓存

# 进入git服务器-->这个操作需要root权限,不然连文件夹都进不去cd /var/opt/gitlab/git-data/repositories#根据项目,进入对应的git项目文件夹# 进入 项目.git文件,就可以看到和本地的.git目录中一样的目目录了#查询git项目大小git count-objects -vH   # 此时还是旧的大小git gc --prune=now  # 清理无效文件git count-objects -vH  # 此时就和本地一样,从库减小了

⚠️清理完之后,每个人一定要删掉之前拉取的项目,重新从git上拉项目。不要使用之前的项目了！！！之前的项目中的.git文件会将已将删除的文件重新加进来,甚至变的更大

参考